Gaming device with rotatably placed cameras

ABSTRACT

A method to identify positions of fingers of a hand is described. The method includes capturing images of a first hand using a plurality of cameras that are part of a wearable device. The wearable device is attached to a wrist of a second hand and the plurality of cameras of the wearable device is disposed around the wearable device. The method includes repeating capturing of additional images of the first hand, the images and the additional images captured to produce a stream of captured image data during a session of presenting the virtual environment in a head mounted display (HMD). The method includes sending the stream of captured image data to a computing device that is interfaced with the HMD. The computing device is configured to process the captured image data to identify changes in positions of the fingers of the first hand.

CLAIM OF PRIORITY

This application is a continuation of and claims the benefit of andpriority, under 35 U.S.C. § 120, to U.S. patent application Ser. No.16/005,496, filed on Jun. 11, 2018, titled “Gaming Device With RotatablyPlaced Cameras”, and issued as U.S. Pat. No. 10,620,711, on Apr. 14,2020, and which is a continuation of and claims the benefit of andpriority, under 35 U.S.C. § 120, to U.S. patent application Ser. No.15/694,511, filed on Sep. 1, 2017, titled “Gaming Device With RotatablyPlaced Cameras”, and issued as U.S. Pat. No. 9,996,166 on Jun. 12, 2018,which is a continuation of and claims the benefit of and priority, under35 U.S.C. § 120, to U.S. patent application Ser. No. 15/472,786, filedon Mar. 29, 2017, titled “Gaming Device With Rotatably Placed Cameras”,and issued as U.S. Pat. No. 9,753,549 on Sep. 5, 2017, all of which arehereby incorporated by reference in their entirety.

The U.S. patent application Ser. No. 15/472,786 is a continuation of andclaims the benefit of and priority, under 35 U.S.C. § 120, to U.S.patent application Ser. No. 14/633,415, filed on Feb. 27, 2015, titled“Gaming Device With Rotatably Placed Cameras”, and issued as U.S. Pat.No. 9,649,558 on May 16, 2017, which is hereby incorporated by referencein its entirety.

The U.S. patent application Ser. No. 14/633,415 claims the benefit ofand priority, under 35 U.S.C. § 119(e), to U.S. Provisional PatentApplication No. 61/953,732, filed on Mar. 14, 2014, and titled “GamingDevice With Rotatably Placed Cameras”, which is hereby incorporated byreference in its entirety.

FIELD

The present invention relates to methods and systems for using rotatablyplaced cameras with a gaming device.

BACKGROUND

In current systems that track a user's hands, a user usually holds acontroller or wears a special glove so that the user views his/her handsrepresented in a virtual or an augmented space. The user holding acontroller has a limited range of hand and finger movements; otherwisehe/she may drop the controller. The user wearing a tracking glove alsoundergoes inconvenience of wearing and removing the glove each timehe/she wishes to see his/her hands, and also experiences reduced hygieneoccurring by placement of his/her fingers inside the glove.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments described in the present disclosure provide systems andmethods for using rotatably placed cameras with a gaming device.

In one embodiment, a user wears a rotatable device on his/her wrists.For example, one wearable device is worn on a wrist of the user andanother wearable device is worn on another wrist of the user. A camerais embedded within each wearable device. The camera of the wearabledevice worn on the left wrist captures image data associated with aright hand of the user. For example, the camera captures image data offingers of the right hand, or of the right hand, or of the wearabledevice worn on the right wrist, or of a portion of the right arm, etc.Similarly, the camera of the wearable device worn on the right wristcaptures image data associated with the left hand of the user. The imagedata is transferred to a game console or another computer system, e.g.,another computing device, etc., for determining a position andorientation of at least a portion of the right arm and determining aposition and orientation of at least a portion of the left arm. Theposition and orientation of the portion of the left arm and/or theposition and orientation of the portion of the right arm is used todetermine a state, e.g., color, texture, position, orientation, shade,shape, etc., of a virtual object to be displayed on a head-mounteddisplay (HMD).

In some embodiments, a user wears a wearable device to manipulate, e.g.,grab, move, push, pull, etc., a virtual object in a virtual reality (VR)or an augmented reality (AR) scene, which is displayed on an HMD or on adisplay device, e.g., a television, a computer, etc. A virtual handwithin a game moves when the user moves his/her hand while wearing thewearable device. Moreover, fingers of a virtual hand in the game movewhen the user moves his/her fingers while wearing the wearable device.Position and/or orientation of the fingers are determined from imagedata captured using the cameras described above to generate the movementof fingers of the virtual hand.

For virtual reality or augmented reality, a certain number ofapplications increase immersion or closely replicate reality if aposition and/or orientation of a body part, e.g., a hand, a foot, afinger, a thumb, a combination of the hand and the finger, a combinationof the hand and the thumb, etc., of the user is determined.

In some embodiments, a number of wearable bands, e.g., a pair of wristbands, a pair of ankle bands, a pair of finger bands, a pair of thumbbands, etc., are provided with cameras to generate image data regardinga position and/or orientation of a body part. For example, a wearableband that is integrated with a camera and worn on a finger of the lefthand of the user generates image data of the right hand of the user. Inan embodiment, a wearable band is integrated with a wearable device,e.g., a wrist watch, a bracelet, etc.

In an embodiment, each wearable band has at least one camera that isable to track the other wearable band and/or the other body part. Imagesthat are captured by the wearable band of the other wearable band and/orthe other body part are used by a processor, e.g., a processor of anHMD, a processor of a game console, etc., to detect position and/ororientation of the user's body parts. For example, a relative positionand/or a relative orientation of wrists of the user are determined fromthe images. The relative position and/or the relative orientation areused by a processor to determine a next game state of a game. Forexample, the relative position and/or the relative orientation of a bodypart is used to determine whether the user moves a game piece, e.g., avirtual tennis ball, a virtual weapon, etc., in a VR or an AR image.

In various embodiments, each wearable band includes a number of cameras,e.g., one camera, two cameras, four cameras, etc., so that the camerasare able to point toward the other body part depending on an orientationof the body part and position of the body part. For example, when aventral part of the left arm faces a ventral part of the right arm,cameras placed around a left wrist of the user face a right wrist of theuser to obtain images of the right wrist and/or of the right hand andcameras placed around the right wrist of the user face the left wrist toobtain images of the left wrist and/or of the left hand.

In several embodiments, cameras of wearable bands have wide angle lensfor a wide field of view, so fewer cameras on each wearable band areused.

In some embodiments, wearable bands are connected via a medium, e.g., awired medium, a wireless medium, etc., with each other, to the HMDand/or to a computing device, e.g., the game console, a desktopcomputer, a laptop computer, a tablet computer, a cell phone, etc.Examples of the medium include Bluetooth, Wi-Fi, universal serial bus(USB), a parallel transfer medium, a serial transfer medium, andEthernet. The wearable bands communicate via the medium with each other,with the HMD and/or with the game console. For example, the wearablebands communicate with each other to exchange synchronizationinformation with each other. Examples of the synchronization informationinclude frame rate of a camera of a wearable device, a rate at whichlight emitters of the wearable device are pulsed, etc.

In various embodiments, a wearable band includes inertial sensors, todetect movement and/or orientation of the body part. The inertialsensors generate resistance indicating signals based on a movement ofthe body part on which the wearable band is worn and provide the signalsto a processor. The processor analyzes the signals to determine aposition and/or an orientation of the body part with respect to an xyzco-ordinate system, which is located on a camera of the wearable device.

In several embodiments, at least one camera on each wrist band isdirected at the other wrist, so that each wrist band tracks the otherwrist and movements of the other hand or fingers of the other hand.

In some embodiments, each wearable band includes markers, e.g., flashinglight emitting diodes (LEDs), or quick response (QR) codes, orreflectors, or patterns, or visible lights, or infrared (IR) lights, ora combination thereof, etc., to enable identification of a location ofthe other body part. For example, the markers and cameras of a wearableband are interspersed with each other to provide an alternatearrangement of the markers and the cameras. The camera on the wearableband generates images of markers on the other wearable band and providesthe images to a processor to determine a position and/or an orientationof the other body part.

In various embodiments, a color of a first wearable band is differentfrom a color of a second wearable band to distinguish a first body parton which the first wearable band is worn from a second body part onwhich the second wearable band is worn. A processor is pre-programmed toassociate a color with the first body part and another color with thesecond body part to separate movements of the two body parts.

In several embodiments, each wearable band includes a light emitter,e.g., a fiber optic light emitter, a diffused fiber optic light emitter,etc., so that each wearable band emits a color. The color is detected bya camera of the other wearable band and/or of an HMD and/or of the gameconsole and/or of a display device, e.g., a television, a computingdevice monitor, etc., to enable a processor to determine and positionand/or an orientation of the body part. As an example, a fiber opticcable is looped around a wearable band or defines a pattern of lightemitters that is viewed by a camera integrated in a wearable band thatis worn on the other body part. The pattern is embodied within imagedata that is provided by the camera via the medium to the processor. Theprocessor, based on the pattern embodied within the image data,determines a position and/or an orientation of the body part (e.g.,fingers, wrist, etc.) as viewed by the camera on the other wearabledevice and/or a camera on the HMD and/or a camera connected to the gameconsole. In this example, the fiber optic cable has openings for escapeof light and each opening acts as a light emitter. As another example,light emitters that emit light are placed around a wearable band.

In some embodiments, the user wears colored wearable bands, and thecolored wearable bands do not include any electronics or cameras. Theuser places his/her hands or wrists over a surface, e.g., a whitesurface, a white mat, a white board, etc., and a camera of an HMD or acamera of the game console generates image data including positions andorientations of the colored body part bands and portions of the arms ofthe user and the image data is used to identify position and/ororientation of the hands or wrists of the user.

In various embodiments, the user places his/her wrists and/or hands overa pad device, e.g., a mat, a surface, a board, etc., that is colored(e.g., green screen, blue screen, etc.), and a camera can track thewrists and/or hands. Examples of the pad device include a mat that isflexible and is rolled.

In various embodiments, a camera is an IR camera. In variousembodiments, some cameras on a wearable band are IR cameras and theremaining cameras are visible light cameras.

In an embodiment, a method to identify positions of fingers of a hand isdescribed. The positions are used to render a virtual hand to bedisplayed in a head mounted display (HMD) when presenting a virtualenvironment in the HMD. The method includes capturing images of a firsthand using a plurality of cameras that are part of a wearable device.The wearable device is attached to a wrist of a second hand and theplurality of cameras of the wearable device are disposed around thewearable device so that the plurality of cameras are distributed aroundthe wrist of the second hand. The method includes repeating capturing ofadditional images of the first hand, the images and the additionalimages captured to produce a stream of captured image data during asession of presenting the virtual environment in the HMD. The methodincludes sending the stream of captured image data to a computing devicethat is interfaced with the HMD. The computing device is configured toprocess the captured image data to identify changes in positions of thefingers of the first hand for rendering the virtual hand in the HMDcorresponding to the changes in the positions of the fingers of thefirst hand.

In one embodiment, a method for identifying positions of hands of a userinteracting with a virtual environment displayed in an HMD is described.The method includes capturing images of a first hand of the user using aplurality of cameras that are part of a first wearable device, which isattachable to a wrist of the first hand. The plurality of cameras of thefirst wearable device is disposed at angular positions around the firstwearable device. The method includes capturing images of a second handof the user using a plurality of cameras that are part of a secondwearable device. The second wearable device is attachable to a wrist ofthe second hand. The plurality of cameras of the second wearable deviceis disposed at angular positions around the second wearable device. Themethod includes continuing the capturing of the images from theplurality of cameras of the first and second wearable devices during asession of interactivity with the virtual environment displayed in theHMD. The images captured by the first wearable device include images ofthe second wearable device and images captured by the second wearabledevice include images of the first wearable device. The method includescapturing additional images of the first wearable device and the secondwearable device using a reference camera. The method includes sendingthe images from the first wearable device, the images from the secondwearable device, and the additional images from the reference camera toa computing device that is interfaced with the HMD. The computing deviceis configured to process the images from the first wearable device toidentify positions of the second hand and process the images from thesecond wearable device to identify positions of the first hand, and thecomputing device uses the reference camera to provide a reference forthe positions of the first and second hands.

In an embodiment, a system includes a first wearable device for wearingon a wrist of a first hand of a user. The first wearable device includesa camera for capturing image data of a second hand of the user. Thefirst wearable device includes a communication device for communicatingthe image data captured using the first wearable device. The systemincludes a game console coupled to the first wearable device. The gameconsole has a console communication device coupled to the communicationdevice of the wearable device for receiving the image data from thecommunication device of the wearable device. The game console includes agame processor coupled to the console communication device foridentifying a position of the second hand of the user from the imagedata captured using the first wearable device. The game processor isconfigured to determine data regarding a state of a virtual object in avirtual environment based on the position of the second hand. Theconsole communication device sends the data regarding the state of thevirtual object. The system includes an HMD coupled to the game console.The HMD includes an HMD communication device coupled to the consolecommunication device for receiving the data regarding the state of thevirtual object from the console communication device. The HMD furtherincludes a processing unit coupled to the HMD communication device fordisplaying the virtual object having the state on a display screen ofthe HMD.

Some advantages of the herein described embodiments include providing aclose-up view of a portion of an arm of a user. The close-up view iscaptured by a camera that is integrated within a wearable device. Theclose-up view provides an accurate position and/or orientation of theportion of the arm. The accurate position and/or orientation are used todetermine a state, e.g., color, texture, shade, shape, position,orientation, etc., of a virtual object in an image.

Also, further advantages of the herein described embodiments includeusing a wearable device that is more hygienic and easier to use than aglove. For example, the wearable device is attached to a wrist of a userand there is no enclosure that surrounds fingers and hand of the user.The lack of enclosure improves hygiene for the user. Moreover, there isa lesser risk of a wearable device falling off when a user makes agesture in which his/her fingers are pointing to a floor on which theuser is standing or sitting. The wearable device is fastened to an armof the user.

Other aspects described in the present disclosure will become apparentfrom the following detailed description, taken in conjunction with theaccompanying drawings, illustrating by way of example the principlesdescribed in the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present disclosure are best understood byreference to the following description taken in conjunction with theaccompanying drawings in which:

FIG. 1A is a diagram of a system to illustrate use of wearable devicesto generate images of the opposite wearable devices and/or oppositehands and/or fingers of the opposite hands of a user, in accordance withone embodiment of the present disclosure.

FIG. 1B is a diagram to illustrate use of multiple cameras to capturesimages of left and right hands of the user, in accordance with oneembodiment of the present disclosure.

FIG. 1C is a diagram used to illustrate capturing of image data whenwearable devices include multiple light emitters, in accordance with oneembodiment of the present disclosure.

FIG. 2A is a diagram of a camera within a wearable device, in accordancewith one embodiment of the present disclosure.

FIG. 2B is a diagram of wearable devices to illustrate cameras thatcapture image data when one hand is behind the other hand of the user,in accordance with one embodiment of the present disclosure.

FIG. 3 is a diagram of a system to illustrate wearable devices thatinclude cameras and emitters, in accordance with one embodiment of thepresent disclosure.

FIG. 4 is a diagram of a system in which a wearable device communicateswith a computer, which further communicates with an HMD, in accordancewith one embodiment of the present disclosure.

FIG. 5 is a diagram of an HMD that includes a camera, in accordance withone embodiment of the present disclosure.

FIG. 6 is a diagram of a camera system to illustrate periodicallyturning off and on of a camera of a wearable device based on a positionof the camera with respect to a wrist of a user, in accordance with oneembodiment of the present disclosure.

FIG. 7A is a diagram of a wearable device that includes a number ofmarkers, in accordance with one embodiment of the present disclosure.

FIG. 7B is a diagram to illustrate use of a position of a wearabledevice to determine an orientation of the wearable device with respectto another wearable device, in accordance with one embodiment of thepresent disclosure.

FIG. 8 is a diagram of multiple wearable devices to illustrate use offiber optic cables and light emitters in wearable devices, in accordancewith one embodiment of the present disclosure.

FIG. 9 is a diagram of a system for illustrating a number of cameraswithin a game console to determine a relative position and/ororientation of hands of a user, in accordance with one embodiment of thepresent disclosure.

FIG. 10 is a diagram illustrating various gestures performed by the userwhile wearing wearable devices, in accordance with one embodiment of thepresent disclosure.

FIG. 11 is a diagram to illustrate two users wearing HMDs and wearabledevices to play games with each other, in accordance with one embodimentof the present disclosure.

FIG. 12 is a diagram of a system in which a camera of a television isused to determine a position and/or orientation of an item with respectto an xyz co-ordinate system, in accordance with one embodiment of thepresent disclosure.

FIG. 13 is a diagram of a system in which ankle devices are worn aroundan ankle of a user, in accordance with one embodiment of the presentdisclosure.

FIG. 14 is a diagram of a system in which a user is wearing wearabledevices around his/her wrist and is wearing ankle devices around his/herankles, in accordance with one embodiment of the present disclosure.

FIG. 15 is a diagram of a system in which a user is using a pad devicewith wearable devices, in accordance with one embodiment of the presentdisclosure.

FIG. 16 is a diagram of a system in which a pad device is overlaid on asurface, in accordance with one embodiment of the present disclosure.

FIG. 17 is a block diagram of a wearable device, in accordance with oneembodiment of the present disclosure.

FIG. 18A is a diagram of an image of a virtual environment that isdisplayed on an HMD to illustrate that both hands of a user are used tocontrol a virtual object that is within an image, in accordance with oneembodiment of the present disclosure.

FIG. 18B is a diagram of an image of a virtual environment that isdisplayed on an HMD to illustrate that one hand of a user is used tocontrol a virtual object and another hand of the user is used to controlanother virtual object, in accordance with one embodiment of the presentdisclosure.

FIG. 19 is an isometric view of an HMD, in accordance with oneembodiment of the present disclosure.

FIG. 20 is a diagram of a system to illustrate an interaction of a userwith a virtual environment by using an HMD and a hand-held controller,in accordance with one embodiment of the present disclosure.

FIG. 21 is an isometric view of another HMD, in accordance with oneembodiment of the present disclosure

FIG. 22 is a diagram used to illustrate access of a virtual environmentvia a computer network, in accordance with one embodiment of the presentdisclosure.

FIG. 23 illustrates a user wearing an HMD to access a virtualenvironment, in accordance with one embodiment of the presentdisclosure.

FIG. 24 is a diagram to illustrate example components of an HMD, inaccordance with one embodiment of the present disclosure.

FIG. 25 illustrates an Information Service Provider architecture, inaccordance with one embodiment of the present disclosure.

DETAILED DESCRIPTION

Systems and methods for using rotatably placed cameras with a gamingdevice are described. It should be noted that various embodimentsdescribed in the present disclosure may be practiced without some or allof these specific details. In other instances, well known processoperations have not been described in detail in order not tounnecessarily obscure various embodiments described in the presentdisclosure.

FIG. 1A is a diagram of an embodiment of a system to illustrate use ofwearable devices 102A and 102B to generate images of the oppositewearable devices 102A and 102B and/or opposite hands and/or fingers ofthe opposite hands of a user. Each wearable device is worn around thecorresponding wrist of the user. For example, a wearable device is awrist band, a watch, a bracelet, a flexible band, a rubber band, etc. Inone embodiment, a wearable device is made of a solid material, e.g., ametal, etc. In another embodiment, a wearable device is made of aflexible material, e.g., fabric, plastic, rubber, etc. The wearabledevice 102A is worn on a left wrist of the user and the wearable device102B is worn on a right wrist of the user.

Each wearable device 102A and 102B includes one or more cameras, e.g.,C1 and C2, etc. The cameras C1 and C2 face each other. For example, thecamera C1 faces a lens of the camera C2 and/or the right hand of theuser. As another example, the camera C2 faces a lens of the camera C1and/or the left hand of the user. As yet another example, the wearabledevice 102A is worn on a wrist of the user so that the camera C1 of thewearable device 102A is located on a ventral side of the left hand. Asanother example, the wearable device 102B is worn on a wrist of the userso that the camera C2 of the wearable device 102B is located on ventralside of the right hand. Examples of a camera include a depth camera, awide field-of-view camera, a digital camera, an infrared camera, etc.

While the user is wearing wearable devices, the user is making gestureswith his/her body part, e.g., a wrist, a hand, a forearm, a leg, anankle, a knee, a finger, a foot, an arm, a finger joint, etc. Forexample, the user moves his/her hand up or down in a real-worldenvironment, e.g., a room in which the user is located, an open space inwhich the user is located, etc. As another example, the user moveshis/her hand left or right or diagonally. As yet another example, theuser moves his/her hand to traverse a virtual environment, e.g., anaugmented reality environment, a virtual reality environment, a gameenvironment, an environment generated from data accessed via a computernetwork, etc. As another example, the user moves his/her hand to changea position and/or orientation of a virtual object in a virtualenvironment. To illustrate, the user makes a triggering gesture withhis/her index finger to press a virtual trigger of a virtual gun in agame. As another illustration, the user makes a side hand motion byextending his/her fingers and moving his/her palm from right to left topush aside a virtual object. Other examples of a virtual object includea virtual vehicle, a virtual hand of the user, a virtual user, a virtualsword, an avatar of the user, a virtual finger, a virtual finger joint,a virtual tree, a virtual flower, etc.

The camera C1 of the wearable device 102A generates images of the rightwrist of the user and/or of the right hand of the user, and/or of thewearable device 102B, and/or of fingers of the right hand of the userand/or of finger joints of one or more fingers of the right hand of theuser. Similarly, the camera C2 of the wearable device 102B generatesimage data of the left wrist of the user and/or of the left hand of theuser, and/or of the wearable device 102A, and/or of fingers of the lefthand of the user and/or of finger joints of one or more fingers of theleft hand of the user. The image data generated by the cameras C1 and C2is communicated via a medium, e.g., a wired medium, a wireless medium,etc., to a head mounted display (HMD) or to a game console.

A processor of the HMD or a processor of the game console analyzes theimage data to determine a position of the right wrist with respect tothe camera C1 worn on the left wrist, and/or a position of fingers ofthe right hand with respect to the camera C1 worn on the left wrist,and/or a position of the right hand with respect to the camera C1 wornon the left wrist, and/or an orientation of the right wrist with respectto the camera C1 worn on the left wrist, and/or an orientation offingers of the right hand with respect to the camera C1 worn on the leftwrist, and/or an orientation of the right hand with respect to thecamera C1 worn on the left wrist.

A processor of the HMD or a processor of the game console analyzes theimage data to determine a position of the left wrist with respect to thecamera C2 worn on the right wrist, and/or a position of fingers of theleft hand with respect to the camera C2 worn on the right wrist, and/ora position of the left hand with respect to the camera C2 worn on theright wrist, and/or an orientation of the left wrist with respect to thecamera C2 worn on the right wrist, and/or an orientation of fingers ofthe left hand with respect to the camera C2 worn on the right wrist,and/or an orientation of the left hand with respect to the camera C2worn on the right wrist.

Examples of a processor include an application specific integratedcircuit (ASIC), a programmable logic device (PLD), a microprocessor, acontroller, a central processing unit, etc.

In some embodiments, a lower number of cameras are used on a wearabledevice if each camera is a wide field of view camera than if each camerais a narrow field of view camera.

In various embodiments, a camera is integrated within a wearable device.In some embodiments, a camera is placed on top of the wearable device.

In an embodiment, a camera is programmed to capture an image of awearable device at regular intervals. For example, a camera is coupledto a processor, e.g., a game processor, a processor of an HMD, etc.,which sends a signal to the camera periodically via one or morecommunication devices. Upon receiving the signal, the camera captures animage of a wearable device.

In one embodiment, a camera includes a processor that is pre-programmedto send a signal to a switch to periodically capture an image of awearable device. The switch controls the camera to capture image data.

In one embodiment, a power source provides power to electronics, e.g., acamera, a processor, a light sensor, a light emitter, etc., within awearable device. The power source is located within the wearable device.

In an embodiment, a camera is embedded within a wearable device and alens of the camera extends outside a surface of the wearable device tofacilitate capturing images of another wearable device. For example, awearable device when worn has an inside surface that abuts the body partof the user and has an outside surface that does not abut the body part.The lens is located on the outside surface when the wearable device isworn. As another example, a wearable device when worn on an arm of theuser has a camera having a lens that faces the other hand of the user.

It should be noted that the camera C1 captures image data of an item,e.g., the right hand of the user, fingers of the right hand of the user,finger joints of the right hand of the user, palm of the right hand ofthe user, the wearable device 102B, ventral side of the right hand,dorsal side of the right hand, etc. The image data is used to determinea position and/or orientation of an item from a reference point, e.g.,an origin (0, 0, 0), etc., of an xyz co-ordinate system and thereference point is on the camera C1. Similarly, the camera C2 capturesimage data of an item, e.g., the left hand of the user, fingers of theleft hand of the user, finger joints of the left hand of the user, palmof the left hand of the user, the wearable device 102A, dorsal side ofthe left hand, ventral side of the left hand, etc. The image data isused to determine a position and/or orientation of an item from areference point, e.g., an origin (0, 0, 0), etc., of the xyz co-ordinatesystem and the reference point is located on the camera C2.

In an embodiment, an orientation of a limb of a user includes an angleformed by an axis that passes through a length of the limb with respectto an x-axis of the xyz co-ordinate system, an angle formed by the axisthat passes through the length of the limb with respect to a y-axis ofthe xyz co-ordinate system, and an angle formed by the axis that passesthrough the length of the limb with respect to a z-axis of the xyzco-ordinate system.

In one embodiment, each wearable device, described herein, includes oneor more inertial sensors to generate a position and/or an orientation ofthe wearable device. For example, the wearable device 102A includes anumber of magnetometers, a number of accelerometers, and/or a number ofgyroscopes to generate a position and/or orientation of the wearabledevice 102A. The position and/or orientation are communicated to thegame console. The game console uses the position and/or orientation toidentify a state, e.g., color, texture, shape, position, orientation,shade, etc., of a virtual object corresponding to the position and/ororientation of the wearable device 102A.

In one embodiment, the wearable device 102A is worn in a manner similarto wearing a wrist watch. For example, the wearable device 102A hasstraps that wrap around each other to surround a wrist of the user. Thestraps wrap around each other using an engagement mechanism, e.g.,magnets situated within each strap, a hook and engage mechanism in whicha hook is attached to one strap and a hole is formed in another strap toengage with the hook, etc.

In an embodiment, the wearable device 102A is worn around a wrist of theuser to fit surround the wrist in a manner similar to wearing a wristband. For example, the wearable device 102A is made of a flexiblematerial that stretches when a hand is inserted within an opening formedbetween inside surfaces of the wearable device 102A to fit the wearabledevice around a wrist of the user.

In one embodiment, the wearable device 102A has two arms and is worn ina manner similar to wearing a bracelet. For example, one of the two armshinges on and with respect to the other arm. A hook and engage mechanismis opened to allow the wearable device 102A to wrap around a wrist ofthe user. Once the wearable device wraps around the wrist of the user, ahook of one arm is engaged to a hole formed within the other arm toengage the hook with the hole to fit the wearable device 102A to thewrist.

In an embodiment, a wrist is a portion of an arm of a user between anelbow and a hand of the arm. In one embodiment, a wrist is a portion ofan arm of a user between an elbow and a hand of the arm and portion iscloser to the hand than to the elbow.

FIG. 1B is a diagram to illustrate use of multiple cameras to capturesimages of left and right hands of the user. A wearable device 105A isattached to, e.g., worn around, fitted around, etc., a wrist of the lefthand of the user and another wearable device 105B is attached to a wristof the right hand of the user. The wearable device 105A is an example ofthe wearable device 102A (FIG. 1A) and the wearable device 105B is anexample of the wearable device 102B (FIG. 1A). The wearable device 105has integrated therein cameras C1, C11, C12, and C13. Similarly, thewearable device 105B has integrated therein cameras C2, C21, C22, andC23.

The cameras C1, C11, C12, and C13 are distributed around the wrist ofthe left hand to be located at angular positions of a1, a2, a3, and a4,and the cameras C2, C21, C22, and C23 are distributed around the wristof the right hand to be located at the angular positions of a1, a2, a3,and a4. For example, the cameras C1, C11, C12, and C13 are spaced apartfrom each other at equal angular positions. To further illustrate, thecamera C1 is located at an angle of 90 degrees with respect to thecamera C11, the camera C11 is located at an angle of 90 degrees withrespect to the camera C12, the camera C12 is located at an angle of 90degrees with respect to the camera C13, and the camera C13 is located atan angle of 90 degrees with respect to the camera C1. As anotherexample, the cameras C1, C11, C12, and C13 are spaced apart from eachother at unequal angular positions. For example, the angle a1 is unequalto one or more of the angles a2, a3, and a4. As another example, thecameras C2, C21, C22, and C23 are spaced apart from each other at equalangular positions. To further illustrate, the camera C2 is located at anangle of 90 degrees with respect to the camera C21, the camera C21 islocated at an angle of 90 degrees with respect to the camera C22, thecamera C22 is located at an angle of 90 degrees with respect to thecamera C23, and the camera C23 is located at an angle of 90 degrees withrespect to the camera C2. As another example, the cameras C2, C21, C22,and C23 are spaced apart from each other at unequal angular positions.

Angular positions of cameras of a wearable device are formed withrespect to lines that pass through a centroid of the wearable device.For example, the angle a1 is formed with respect to a horizontal dashedline that passes through a centroid CTD1 of the wearable device 105A,the angle a2 is formed with respect to a vertical line that passesthrough the centroid CTD1, the angle a3 is formed with respect to thehorizontal line, and the angle a4 is formed with respect to the verticalline.

The cameras C1, C11, C12, and C13 capture image data of the right handof the user and the cameras C2, C21, C22, and C23 capture image data ofthe left hand of the user. For example, when the wearable device 105B isworn by the user on the wrist of his/her right hand, the camera C2captures image data of the left hand of the user. When the wearabledevice 105B turns, e.g., rotates, etc., with respect to the wrist of theright hand of the user during interaction of the user with a virtualenvironment, the camera C21 instead of the camera C2 faces the left handof the user and captures image data of the left hand. As anotherexample, when the wearable device 105A is worn by the user on the wristof his/her left hand, the camera C1 captures image data of the righthand of the user. When the wearable device 105A turns with respect tothe wrist of the left hand of the user during interaction of the userwith the virtual environment, the camera C11 instead of the camera C1faces the right hand of the user and captures image data of the righthand.

In one embodiment, instead of a wearable device, e.g., the wearabledevice 105B, the wearable device 105A, etc., turning with respect to awrist of a hand on which the wearable device is worn, the wearabledevice is fitted, e.g., by pressing, by using a wrap-around belt, byusing a hook and engage mechanism, by using a strap, etc., to the wristto reduce chances of the wearable device turning with respect to thewrist of the right hand of the user. For example, before the wrist ofthe right hand is turned, the camera C2 captures image data of the lefthand of the user and after the wrist is turned, e.g., clockwise, etc.,the camera C23 captures image data of the left hand of the user. Asanother example, before the wrist of the left hand is turned, the cameraC1 captures image data of the right hand of the user and after the wristis turned, e.g., counterclockwise, etc., the camera C13 captures imagedata of the right hand of the user.

It should be noted that although each wearable device is shown asincluding four cameras, in one embodiment, each wearable device includesmore or less than four cameras. For example, the wearable device 105Aincludes six cameras that are equally or unequally spaced apart.

In an embodiment, the wearable device 105A includes a different numberof cameras than that included within the wearable device 105B.

In an embodiment, a wearable device includes a processor that isconnected a camera of the wearable device to receive image data that iscaptured by the camera. The processor of the wearable device is embeddedwithin the wearable device, e.g., is located within a hollow spacewithin a housing of the wearable device, etc. The processor of thewearable device determines whether a hand of the user is visible inimage data that is received from the camera. For example, the processorof the wearable device determines whether pixel data that is a part ofthe image data includes a shape of a hand, or a texture of the hand, ora color of the hand, or a combination of two or more thereof. Upondetermining that the pixel data includes the shape or the texture or thecolor or the combination of two or more thereof, the processordetermines that the image data includes an image of the hand. On theother hand, upon determining that the pixel data does not include theshape, the texture, and/or the color, the processor determines that theimage data does not include the image of the hand. The processor isconnected to a switch, e.g., a transistor, a group of transistors, etc.,that connects the camera to a power supply, e.g., a battery, etc., thatis also embedded within the hollow space of the wearable device. Upondetermining that the image data does not include the image of the hand,the processor turns off the switch to remove power from being suppliedfrom the power supply to the camera to turn off the camera. On the otherhand, upon determining that the image data includes the image of thehand, the processor continues to maintain the switch in an on positionto continue a supply of power from the power supply to the camera tocontinue capturing of image data by the camera.

In an embodiment, instead of turning off a camera upon determining thatthe image data does not include an image of a hand of a user, aprocessor of a wearable device sends a signal to the camera of thewearable device to reduce a frame rate with which images are captured bythe camera and/or to reduce a resolution of images captured by thecamera. Upon determining that image data captured by the camera includesthe image of the hand after sending the signal to reduce the resolution,the processor sends another signal to the camera to increase theresolution of images to a pre-determined amount. Similarly, upondetermining that image data captured by the camera includes the image ofthe hand after sending the signal to reduce the frame rate, theprocessor sends another signal to the camera to increase the frame rateto a pre-determined level.

In one embodiment, instead of the processor being located in thewearable device to determine whether to turn on or off a camera and/orto decrease the frame rate and/or to decrease the resolution based onwhether image data includes an image of a hand, the processor is locatedin a game console. Image data that is captured by the camera of thewearable device is provided via a communication device of the wearabledevice and a communication device of the game console to a gameprocessor of the game console. The game processor makes the samedetermination as that described above as being made the processor of thewearable device and provides the determination to the wearable device tocontrol a switch that is located in the wearable device viacommunication devices of the game console and of the wearable device.

In one embodiment, a processor determines from image data captured usinga camera of a wearable device that the camera remain on to capturefurther image data and determines from image data captured using anothercamera of the wearable device that the other camera be turned off. Forexample, upon determining from image data captured using the camera C1of the wearable device 105A that the camera C1 is oriented to face theright hand of the user and upon determining from image data capturedusing the camera C12 of the wearable device 105A that the camera C12 isoriented to face away from the right hand of the user, a processor ofthe wearable device or of a game console determines that the camera C1remain on and that the camera C12 be turned off. To illustrate, a gameprocessor determines that the camera C1 is oriented to face the righthand of the user when image data generated by the camera C1 includes animage of the right hand. Such turning on and off of cameras saves powerand also reduces image data that is generated by the cameras to reducechanges of information overload for analysis.

FIG. 1C is a diagram used to illustrate capturing of image data whenwearable devices 107A and 107B include multiple light emitters. Forexample, the wearable device 107A includes light emitters LES1, LES11,LES12, and LES 13 distributed on a surface of the wearable device 107A.The light emitters LES1, LES11, LES12, and LES 13 are distributed on thesame surface on which the cameras C1, C11, C12, and C13 are distributed.As another example, the wearable device 107B includes light emittersLES2, LES21, LES22, and LES23 distributed on a surface of the wearabledevice 107B. The light emitters LES2, LES21, LES22, and LES 23 aredistributed on the same surface on which the cameras C2, C21, C22, andC23 are distributed. The wearable device 107A is an example of thewearable device 102A (FIG. 1A) and the wearable device 107B is anexample of the wearable device 102B (FIG. 1A). The wearable device 107Ais attached to, e.g., worn on, surrounds, fitted to, etc., the leftwrist of the user and the wearable device 107B is attached to the rightwrist of the user.

Light emitters of a wearable device are equally or unequally spacedapart on a surface of the wearable device. For example, the lightemitter LES1 forms an angle with respect to the light emitter LES13 andthe light emitter LES13 forms the same angle with respect to the lightemitter LES12. Also, in this example, the light emitter LES11 forms thesame angle with respect to the light emitter LES12 and the light emitterLES1. As another example, the light emitter LES1 forms a first anglewith respect to the light emitter LES13 and the light emitter LES13forms a second angle with respect to the light emitter LES12. Also, inthis example, the light emitter LES11 forms a third angle with respectto the light emitter LES12 and a fourth angle with respect to the lightemitter LES1. In this example, the first angle is different from atleast one of the second, third, and fourth angles.

Light emitters of a wearable device are interleaved, e.g., interspersed,etc., with cameras of the wearable device. For example, the lightemitter LES1 is located between the cameras C1 and C11, the lightemitter LES11 is located between the cameras C11 and C12, the lightemitter LES12 is located between the cameras C12 and C13, and the lightemitter LES13 is located between the cameras C13 and C1. Similarly, thecamera C1 is located between the light emitters LES1 and LES13, thecamera C11 is located between the light emitters LES1 and LES11, thecamera C12 is located between the light emitters LES11 and LES 12, andthe camera C13 is located between the light emitters LES12 and LES13. Asanother example, the light emitter LES2 is located between the camerasC21 and C22, the light emitter LES21 is located between the cameras C21and C2, the light emitter LES22 is located between the cameras C2 andC23, and the light emitter LES23 is located between the cameras C22 andC23. Similarly, the camera C2 is located between the light emittersLES21 and LES22, the camera C23 is located between the light emittersLES22 and LES23, the camera C22 is located between the light emittersLES23 and LES2, and the camera C21 is located between the light emittersLES2 and LES21.

Any of cameras C2, C21, C22, and C23 detect light emitted by one or moreof the light emitters LES1, LES11, LES12, and LES13 that are within afield-of-view of the camera to capture image data that includespositions and orientations of the wearable device 107A. For example,when the light emitter LES1 is oriented with respect to the left wristof the user to direct light towards the camera C2 while the camera C2 isoriented on the right wrist of the user to face the LES1, the camera C2captures light emitted by the light emitter LES1. Similarly, any ofcameras C1, C11, C12, and C13 detect light emitted by one or more of thelight emitters LES2, LES21, LES22, and LES23 that are within afield-of-view of the camera to capture image data that includespositions and orientations of the wearable device 107B. For example,when the light emitter LES2 is oriented with respect to the right wristof the user to direct light towards the camera C1 while the camera C1 isoriented on the left wrist of the user to face the light emitter LES2,the camera C1 captures light emitted by the light emitter LES2.

In one embodiment, an LES of a first wearable device is oriented withrespect to a wrist of the user to direct light towards a camera of asecond wearable device worn on another wrist of the user after the firstwearable device turns, e.g., rotates clockwise, rotatescounterclockwise, etc., with respect to the wrist on which the firstwearable device is worn.

In an embodiment, an LES of a first wearable device is oriented withrespect to the a first wrist of the user to direct light towards acamera of a second wearable device worn on a second wrist of the userafter the first wrist and/or the second wrist is turned. In thisembodiment, a position of the LES of the first wearable device withrespect to the first wrist is constant is substantially constant and aposition of the LES of the second wearable device with respect to thesecond wrist is constant or is substantially constant. A position of awearable device with respect to a wrist is constant when the wearabledevice is fitted to the wrist.

In an embodiment, a light emitter and/or a camera are attached, e.g.,integrated within, fitted within, glued to, soldered to, embeddedwithin, etc., to a wearable device.

In one embodiment, a wearable device includes more or less than fourlight emitters.

In one embodiment, a light emitter of a wearable device is constantlyemitting light.

In one embodiment, a light emitter of a wearable device is emittinglight at a frequency, e.g., is strobing, is pulsing, is flashing, etc.For example, light emitters of a wearable device emit light in around-robin fashion. To illustrate, the light emitter LES1 pulses onceto emit light first, the light emitter LES13 then pulses once to emitlight second, the light emitter LES12 then pulses once to emit lightthird, the light emitter LES11 pulses once to emit light fourth, and thelight emitter LES1 pulses once to emit light to continue the round-robinfashion, e.g., a counter-clockwise fashion, etc., of light emission. Inthis illustration, when a light emitter of the wearable device 107A isturned on, all remaining light emitters of the wearable device 107A areturned off. As another illustration, the light emitters LES1, LES11,LES12, and LES13 emit light in a clock-wise round-robin fashion.

In an embodiment, a light emitter of a wearable device emits light of adifferent color than another light emitter of the wearable device. Forexample, the light emitter LES1 emits light of a blue color, the lightemitter LES11 emits light of a green color, the light emitter LES12emits light of a red color, and the light emitter LES13 emits light of awhite color. Moreover, in this embodiment, the wearable device is fittedto a hand of the user so as to not be disoriented during movement of thehand. For example, the light emitter LES13 is positioned to be on aventral side of the left hand of the user, the light emitter LES11 ispositioned to be on a dorsal side of the left hand, the light emitterLES1 is positioned to be on a first lateral side of the left hand, andthe light emitter LES12 is positioned to be on a second lateral side ofthe left hand. The difference in colors facilitates an identification ofan orientation of a hand of the user on which a wearable device emittingthe different colors is worn. For example, a game processor of a gameconsole determines from image data captured by a camera that a hand ofthe user is oriented so that a ventral side of the hand faces thecamera. The image data includes a white color of light that is emittedby the light emitter LES13. As another example, a game processor of agame console determines from image data captured by a camera that a handof the user is oriented so that a dorsal side of the hand faces thecamera. The image data includes a green color of light that is emittedby the light emitter LES11.

In one embodiment, a device, e.g., a light emitter, a camera, ispositioned on a side, e.g., dorsal side, lateral side, ventral side,etc., of an arm when the device is located over or under the side or toone side of the side and also is adjacent to the side of the arm. Forexample, a light emitter is located on a dorsal side of a wrist that isturned to be upside-down when the light emitter is located under thedorsal side and is located adjacent to the dorsal side. As anotherexample, a light emitter is located on a dorsal side of a wrist that isturned to a lateral-side-up position when the light emitter is locatedto a side of the dorsal side and is located adjacent to the dorsal side.

In an embodiment, some light emitters of a wearable device areconstantly emitting light and the remaining light emitters of thewearable device are emitting light at a frequency.

In one embodiment, the wearable devices 107A and 107B exchangesynchronization information. For example, the wearable device 107Aincludes a processor that controls a frame rate with which one or morecameras of the wearable device 107A captures images and sends the framerate via a communication device of the wearable device 107A to acommunication device of the wearable device 107B using a wired or awireless communication protocol. A processor of the wearable device 107Breceives the frame rate and controls one or more cameras of the wearabledevice 107B to achieve the frame rate. As another example, a processorof the wearable device 107A controls a frequency of emission of light bythe light emitters of the wearable device 107A. A communication deviceof the wearable device 107A is coupled to the processor and sends thefrequency to a communication device of the wearable device 107B using awired or a wireless communication protocol. Upon receiving thefrequency, a processor of the wearable device 107B controls lightemitters of the wearable device 107B to emit light at the frequency.

FIG. 2A is a diagram of an embodiment of a camera C3 within a wearabledevice 103A to illustrate capture of image data of a dorsal side or aventral side of a hand of the user. The camera C3 is attached to, e.g.,soldered to, glued to, etc., an edge of the wearable device 103A toobtain a view of a dorsal portion of a hand of the user. Moreover, inFIG. 2A, another camera C4 within the wearable device 103B is attachedto an edge of the wearable device 103B to obtain a view of a bottom,e.g., ventral portion, palm, etc., of the hand of the user.

It should be noted that the camera C3 has a field-of-view (FOV) tofacilitate capturing image data of an item, e.g., the left hand of theuser, fingers of the left hand of the user, finger joints of the lefthand, dorsal part of the left hand of the user, etc. The image datacaptured by the camera C3 is used to determine a position and/ororientation of an item from a reference point, e.g., an origin (0, 0,0), etc., of the xyz co-ordinate system and the reference point is onthe camera C3. Similarly, the camera C4 has an FOV to facilitatecapturing image data of an item, e.g., the right hand of the user,fingers of the right hand of the user, finger joints of fingers of theright hand, palm of the right hand of the user, etc. The image datacaptured by the camera C4 is used to determine a position and/ororientation of an item from a reference point, e.g., an origin (0, 0,0), etc., of the xyz co-ordinate system and the reference point islocated on the camera C4.

FIG. 2B is a diagram of an embodiment of wearable devices 109A and 109Bto illustrate cameras C31 and C41 that capture image data when one handis behind the other hand of a user. For example, a field-of-view of thecamera C31 is behind the left hand of the user to capture image data ofthe right hand when the right hand is behind the left hand. As anotherexample, a field-of-view of the camera C41 is behind the right hand ofthe user to capture image data of the left hand when the left hand isbehind the right hand. It should be noted that the camera C31 is locatedon a dorsal side of the left hand of the user and the camera C41 islocated on a ventral side of the right hand of the user. The wearabledevice 109A is an example of the wearable device 103A (FIG. 2A) and thewearable device 109B is an example of the wearable device 103B (FIG.2A).

It should be noted that the camera C31 is located on an edge of thewearable device 109A that is opposite to an edge on which the camera C3is located. For example, the camera C3 is located on a front edge of thewearable device 109A and the camera C31 is located on a back edge of thewearable device 109A. As another example, the camera C3 has afield-of-view in a direction opposite to a direction of field-of-view ofthe camera C31. Similarly, the camera C41 is located on an edge of thewearable device 109B that is opposite to an edge on which the camera C4is located.

In one embodiment, the wearable device 109A includes any number ofcameras located at edges of the wearable device 109A. For example, thewearable device 109A includes cameras that are located at the back edgeof the wearable device 109A and that are adjacent to lateral sides ofthe left arm of the user. Similarly, the wearable device 109B includesany number of cameras located at edges of the wearable device 109B.

The image data captured by the camera C31 is used to determine aposition and/or orientation of an item from a reference point, e.g., anorigin (0, 0, 0), etc., of the xyz co-ordinate system and the referencepoint is located on the camera C31. Similarly, the image data capturedby the camera C41 is used to determine a position and/or orientation ofan item from a reference point, e.g., an origin (0, 0, 0), etc., of thexyz co-ordinate system and the reference point is located on the cameraC41.

FIG. 3 is a diagram of an embodiment of a system to illustrate wearabledevices 104A and 104B that include cameras and emitters. The wearabledevice 104A is an example of the wearable device 102A and the wearabledevice 102A is an example of the wearable device 104B.

Each wearable device 104A and 104B includes an arrangement of camerasand light emitters, e.g., light emitting diodes, infrared light emitter,incandescent lamps, gas discharging lamps, etc. For example, thewearable device 104B has an embedded light emitter LE1 and anotherembedded light emitter LE2. As another example, the wearable device 104Ahas an embedded light emitter LE3 and another embedded light emitterLE4. The wearable device 104A and 104B are connected via a wired or awireless medium, e.g., a conductor, a cord, radio frequency signals,etc., to a game console 106.

A light emitter of a wearable device, e.g., the wearable device 104A,etc., emits light, e.g., visible light, infrared light, etc., towards acamera of another wearable device, the wearable device 104B, etc. Acamera of the wearable device 104A generates image data based on thelight that is reflected from the right hand of the user on which thewearable device 104B is worn. The image data is transferred via a mediumto the game processor of the game console 106. Based on the image data,the game processor of the game console 106 determines a relativeorientation and/or relative position of the right hand of a user 302with respect to the left hand of the user 302 and uses the relativeorientation and/or the relative position to determine a gesture made bythe user 302. The light emitted from a light emitter of a wearabledevice that is worn on an arm of the user 302 facilitates identificationby the game processor of the game console 106 of a position and/ororientation of the hand of the user 302. A state of a virtualenvironment that is displayed on a television 108 or on an HMD 310 ischanged to correspond to the gesture and the change in the state istransferred from the game console 106 to the television 108 and/or tothe HMD 310. A processor of the television 108 renders a virtualenvironment on a display screen of the television and/or a processor ofthe HMD 310 renders the virtual environment on a display screen of theHMD 310 based on the change in the game state.

In various embodiments, image data generated by a camera is transferredvia a medium to the HMD 310 or the game console 106. Based on the imagedata, the processor of the HMD 310 or the game processor of the gameconsole 106 determines a relative orientation and/or relative positionof the left hand of the user 302 with respect to a camera on the righthand of the user and uses the relative orientation and/or the relativeposition to determine a gesture made by the user 302. The gesture isused by the game processor of the processor of the HMD 310 to identify astate of a virtual object. The state of the virtual object iscommunicated from the game console 106 to the HMD 310 via a wired or awireless medium. The state of the virtual object is used to change achange of the virtual object that is displayed on the HMD.

In one embodiment, a wearable device includes any number of lightemitters.

In an embodiment, a light emitter is an example of a marker.

FIG. 4 is a diagram of an embodiment of a system in which a wearabledevice communicates with the game console 106, which furthercommunicates with the HMD 310. Each wearable device 102A and 102B isconnected via a medium, e.g., an Ethernet medium, a Wi-Fi medium, awireless connection, a wired connection, a Bluetooth connection, auniversal serial bus (USB) connection, etc., to the game console 106.Image data of the right hand is transferred from the wearable device102A via a medium to the game console 106 and image of the left hand istransferred from the wearable device 102B via a medium to the gameconsole 106. The game console 106 includes the game processor thatprocesses the image data to determine positions and/or orientations ofthe hands of the user 302 with respect to each other. The positionsand/or the orientations are used to identify a gesture and a manner inwhich the gesture affects a state of a virtual object in a virtualenvironment, e.g., video conferencing environment, game environment,augmented reality image, virtual reality image, etc. Data regarding thestate is sent to the HMD 310 for display of a virtual object having thestate on a display screen of the HMD 310.

In various embodiments, a camera 402 views the HMD 310 and the wearabledevices 102A and 102B to generate image data of the HMD 310 and thewearable devices 102A and 102B. The image data is provided to the gameconsole 106 for determining the positions and/or orientations of thehands of the user 302 and for determining a position and/or anorientation of the HMD 310. The image data that includes positions andthe orientations of the hands of the user 302 and that includes theposition and/or orientation of the HMD 310 is sent via a communicationdevice of the camera 402 and a communication device of the game console106 to the game processor of the game console 106. The game processor ofthe game console 106 processes the image data to obtain the positionsand orientations of the hands of the user 302 and to obtain the positionand orientation of the HMD 310. The game processor identifies from thepositions and orientations of the hands a gesture performed by the user302, and further identifies from the gesture a state of a virtualobject. Moreover, the game processor identifies from the position andorientation of the HMD 310 and the positions and orientations of thehands of the user 302, relative positions and relative orientationsbetween the hands of the user 302 and the HMD 310. The relativepositions and the relative orientations are used by the game processorto identify a state of a virtual object. Data regarding the state of thevirtual object is sent by the game processor via a communication deviceof the game console 106 and a communication device of the HMD 310 to aprocessor of the HMD 310. The processor of the HMD 310 displays thevirtual object having a state on a display screen of the HMD 310.

It should be noted that image data generated by the camera 402 is from areference point, e.g., origin (0, 0, 0), etc., of the xyz co-ordinatesystem, and the reference point is located at a point on the camera 402,e.g., a point on a lens of the camera 402, etc.

In one embodiment, a position and orientation of an item of the user 302determined from image data captured using the camera 402 is used by thegame processor of the game console 106 to confirm or deny an accuracy ofa position and orientation of the item determined from image datacaptured using a camera of a wearable device. For example, the gameprocessor converts image data captured using the camera of the wearabledevice to be relative to the xyz co-ordinate system located at thecamera 402 instead of being relative to the xyz co-ordinate systemlocated at the camera of the wearable device. To illustrate, the gameprocessor adds respective x, y, and z distances between the xyzco-ordinate system located at the camera 402 and the xyz co-ordinatesystem located at the camera of the wearable device to x, y, and zdistances of the item as viewed by the camera of the wearable device togenerate converted positions. As another illustration, the gameprocessor adds angles formed between the respective x, y, and z axes ofthe xyz co-ordinate system located at the camera 402 and respective x,y, and z axes of the xyz co-ordinate system located at the camera of thewearable device to angles formed by respective axes of the item asviewed by the camera of the wearable device to generate convertedorientations.

Upon determining that the converted position and converted orientationof the item of the user 302 determined from image data captured usingthe camera of the wearable device is accurate, the game processor 106identifies from the position and orientation determined from the imagedata captured using the camera 402 or from the image data captured usingthe camera of a wearable device, a state of a virtual object. On theother hand, upon determining that the converted position and convertedorientation of the item of the user 302 determined from image datacaptured using the camera of the wearable device is not accurate, thegame processor of the game console 106 waits for additional image datafrom the wearable device and additional image data from the camera 402to determine whether a converted position and converted orientation ofan item of the user 302 determined from the additional image datacaptured by the wearable device is accurate compared to a position andorientation of the item determined from the additional image datacaptured by the camera 402.

In one embodiment, upon determining that the converted position andconverted orientation of the item of the user 302 is not accurate, thegame processor of the game console 104 identifies a state of a virtualobject from a correspondence, e.g., mapping, association, link, etc.,between the state and a position and orientation of the item determinedfrom image data captured using the camera 402.

FIG. 5 is a diagram of an embodiment of an HMD 510 that includes acamera 512. For example, the camera 512 is integrated, e.g., embeddedinto, fitted within, situated within, etc., a compartment within the HMD510 so that a lens of the camera 512 can view a portion of thereal-world environment that is in front of the camera 512. In anembodiment, the HMD 510 is an example of the HMD 310 (FIG. 4). Thecamera 512 of the HMD 510 generates image data of an item, e.g., thehands of the user 302 including the wrists of the user and/or fingers ofthe user, palms of the user 302, wearable devices 102A and 102B worn bythe user 302, etc.

A communication device of the HMD 510 communicates, using a wired or awireless communication protocol, the image data to the game processor ofthe game console 106. The game processor of the game console 106determines a position and orientation of an item from a position andorientation of the item in the image data.

In an embodiment, a position and orientation of the item determined fromimage data captured using the camera 512 of the HMD 510 is used toconfirm or deny an accuracy of a position and orientation of the itemdetermined from image data captured using a camera of a wearable device.For example, upon determining that a converted position of an item thatis determined from image data received from a camera of the wearabledevice 102A is within a pre-determined distance, e.g., (x, y, z)co-ordinate, etc., of a position of the item that is determined fromimage data received from the camera 512 of the HMD 510, the gameprocessor of the game console 106 confirms an accuracy of the positionof the item determined from the image data captured using the camera ofthe wearable device 102A. As another example, upon determining that aconverted orientation of an item that is determined from image datareceived from a camera of the wearable device 102A is withinpre-determined ranges, e.g., an angle with respect to the x-axis, anangle with respect to the y-axis, and an angle with respect to thez-axis, etc., of orientation of the item that is determined from imagedata received from the camera 512 of the HMD 510, the game processor ofthe game console 106 confirms an accuracy of the orientation of the itemdetermined from the image data captured using the camera of the wearabledevice 102A. As another example, upon determining that a convertedposition of an item determined from image data received from a camera ofthe wearable device 102A is not within a pre-determined distance, e.g.,(x, y, z) co-ordinate, etc., of a position of the item that isdetermined from image data received from the camera 512 of the HMD 510,the game processor of the game console 106 determines that the positionof the item determined from the image data captured using the camera ofthe wearable device 102A is inaccurate. As another example, upondetermining that a converted orientation of an item that is determinedfrom image data received from a camera of the wearable device 102A isnot within a pre-determined range, e.g., an angle with respect to thex-axis, an angle with respect to the y-axis, or an angle with respect tothe z-axis, etc., of orientation of the item that is determined fromimage data received from the camera 512 of the HMD 510, the gameprocessor of the game console 106 determines that the orientation of theitem determined from the image data captured using the camera of thewearable device 102A is not accurate.

It should be noted that a converted position and a converted orientationis determined from image data captured using a camera of a wearabledevice in a manner similar to that described above. For example, thegame processor converts image data captured using the camera of thewearable device to be relative to the xyz co-ordinate system located atthe camera 512 instead of being relative to the xyz co-ordinate systemlocated at the camera of the wearable device. To illustrate, the gameprocessor adds respective x, y, and z distances between the xyzco-ordinate system located at the camera 512 and the xyz co-ordinatesystem located at the camera of the wearable device to x, y, and zdistances of the item as viewed by the camera of the wearable device togenerate converted positions. As another illustration, the gameprocessor adds angles formed between the respective x, y, and z axes ofthe xyz co-ordinate system located at the camera 512 and respective x,y, and z axes of the xyz co-ordinate system located at the camera of thewearable device to angles formed by respective axes of the item asviewed by the camera of the wearable device to generate convertedorientations.

Upon confirming the accuracy, the game processor of the game console 106identifies from the position and/or orientation of an item determinedfrom image data captured using the HMD 510 or using a camera of awearable device data regarding a state of a virtual object, e.g., avirtual football, a virtual vehicle, a virtual weapon, a virtual tree,etc., to be displayed on a display screen of the HMD 510. The gameprocessor sends via a communication device of the game console 106 and acommunication device of the HMD 510, data regarding the state of avirtual object to the HMD 510. The processor of the HMD 510 receives thedata regarding the state of the virtual object and renders the data todisplay the virtual object on a display screen of the HMD 510.

Upon determining that a position and/or orientation of the itemdetermined from the image data captured using the camera of the wearabledevice 102A is not accurate, the game processor of the game processor106 waits until the position and/or orientation is determined to beaccurate from additional image data that is captured using a camera of awearable device and from additional image data that is captured usingthe camera 512 of the HMD 510.

In one embodiment, upon determining that a position and/or orientationof the item determined from the image data captured using the camera ofthe wearable device 102A is not accurate, instead of using a positionand/or orientation of an item determined from image data captured usinga camera of a wearable device, the game processor of the game console106 uses a position and/or orientation of an item determined from imagedata captured using the camera 512 of the HMD 510 to identify dataregarding a state of a virtual object. The data regarding identifiedstate is provided to the HMD 510 to display a virtual object having thestate on the HMD 510.

In one embodiment, a game processor applies a statistical calculation,e.g., calculating average value, etc., to a position determined fromconverted image data that is generated from image data captured using acamera of a wearable device and a position determined from image datacaptured using another camera, e.g., a camera of another wearabledevice, a camera of an HMD, a camera of a television, anindependently-located camera, a camera of a game console, etc. Theconverted image data is generated by the game processor by convertingimage data received from the camera of the wearable device to bepositioned with respect to the other camera in a manner describedherein. The statistical calculation is performed to generate astatistical value of a position and the statistical value is used by thegame processor to identify a state of a virtual object.

In an embodiment, a game processor applies a statistical calculation,e.g., calculating average value, etc., to an orientation determined fromconverted image data that is generated from image data captured using acamera of a wearable device and an orientation determined from imagedata captured using another camera, e.g., a camera of a another wearabledevice, a camera of an HMD, a camera of a television, anindependently-located camera, a camera of a game console, etc. Theconverted image data is generated by the game processor by convertingimage data received from the camera of the wearable device to beoriented with respect to the other camera in a manner described herein.The statistical calculation is performed to generate a statistical valueof an orientation and the statistical value is used by the gameprocessor to identify a state of a virtual object.

In an embodiment, the HMD 510 includes a light emitter, e.g., a visiblelight emitter, an infrared light emitter, etc., that emits light towardsthe wearable devices 102A and 102B. Light that is reflected from thewearable devices 102A and 102B is sensed by a sensor, e.g., visiblelight sensor, infrared light sensor, etc., of the HMD 510 to generateimage data including positions and orientations of the hands of theuser. The image data is communicated from a communication device of theHMD 510 to a communication device of the game console 106. The gameprocessor parses the image data to obtain relative positions andrelative orientations of the head of the user with respect to each handof the user 302. It should be noted that in this embodiment, a positionof the hand of an arm of the user 302 is the same as a position of awearable device that is worn on the arm and an orientation of the arm isthe same as an orientation of the wearable device.

It should be noted that image data generated by the camera 512 is from areference point, e.g., origin (0, 0, 0), etc., of the xyz co-ordinatesystem, and the reference point is located at a point on the HMD 510,e.g., a point on a lens of the camera 512, etc.

In one embodiment, the camera 512 is an infrared camera that detectsinfrared light. Moreover, each wearable device 102A and 102B includesinfrared light emitters. The infrared light emitters of the wearabledevice 102A emit light towards a hand on which the wearable device 102Bis worn and the infrared light emitters of the wearable device 102B emitlight towards a hand on which the wearable device 102A is worn. Thelight is reflected from hands of the user to be detected by the infraredcamera. The camera 512 generates image data that includes infraredimages.

In an embodiment, an infrared light emitter of a wearable device isdirected towards the hand on which the wearable device is worn to emitlight at the hand instead of being directed towards the other hand of auser.

In one embodiment, infrared light emitters of a wearable device worn ona first arm of a user are pulsed to emit light at a frequency andinfrared light emitter of a wearable device worn on a second arm of theuser are not pulsed, e.g., emit light continuously, etc. A gameprocessor of a game console identifies from image data that includesimages of pulsed infrared light emitters that the wearable device havingthe pulsed emitters provides a position and/or orientation of the firstarm and identifies from image data that includes images of a non-pulsed,e.g., continuously emitting light, etc., that the wearable device havingthe non-pulsed emitters provide a position and/or orientation of thesecond arm.

In an embodiment, instead of distinguishing between pulsed andnon-pulsed infrared light emitters, the game processor identifieswhether a wearable device is attached to the left arm or the right armfrom a frequency of emission of light by infrared light emitters. Inthis embodiment, the infrared light emitters of the wearable deviceattached to the first arm emit light at a different frequency than afrequency of emission of light by the infrared light emitter of thewearable device attached to the second arm.

In one embodiment, the HMD 510 includes any number of cameras, e.g., onecamera for detecting infrared light, another camera for detectingvisible light, all cameras detecting the same type of light, etc.

In an embodiment, a game console or a television includes any number ofcameras, e.g., one camera for detecting infrared light, another camerafor detecting visible light, all cameras detecting the same type oflight, etc.

In one embodiment, instead of one independently-located, any number ofindependently-located cameras is used. For example, oneindependently-located camera detects visible light and anotherindependently located camera detects infrared light. As another example,both independently-located cameras detect the same type of light, e.g.,light having the same wavelength, etc.

FIG. 6 is a diagram of an embodiment of a camera system to illustrateperiodically turning off and on of the camera C1 of the wearable device102A based on a position of the camera C1 with respect to a side of awrist of the user 302. For example, when the camera C1 of the wearabledevice 102A is located on a ventral side 602 of the left hand of theuser 302, the camera C1 is turned on. A determination whether the cameraC1 of the wearable device 102A is located on a ventral side of the lefthand of the user 302 is made by the game processor of the game console106 based on image data that is captured by the camera C2 of thewearable device 102B, which is worn on the right wrist of the user 302.Upon determining that the camera C1 is adjacent to the ventral side ofthe left hand of the user 302, the game processor sends a signal via acommunication device of the game console 106 and a communication deviceof the wearable device 102A to a switch, e.g., a transistor, a group oftransistors, a toggle switch, etc., of the wearable device 102A. Theswitch connects a power supply, e.g., a battery, etc., of the wearabledevice 102A to the camera C1. The signal is used to close the switch tofacilitate provision of power from the power supply to the camera of thewearable device 102A to facilitate capturing images of an item.

As another example, when the camera C1 of the wearable device 102A islocated on a non-viewing side of the left hand of the user 302, e.g., ona dorsal side 604 of the left hand of the user 302, on a lateral side ofthe left hand, etc., so as to not have a field-of-view of the right handof the user 302, the camera C1 is turned off. To illustrate, the cameraC1 is located on a viewing side of the left hand when the camera C1 islocated on a ventral side of the left hand of the user 302, asillustrated in FIG. 1A. A determination whether the camera C1 of thewearable device 102A is located on the non-viewing side of the left handof the user 302 is made by the game processor of the game console 106based on image data that is captured by the camera C2 of the wearabledevice 102B. Upon determining that the camera C1 is located on thenon-viewing side of the left hand of the user 302, the game processorsends a signal via a communication device of the game console 106 and acommunication device of the wearable device 102A to the switch of thewearable device 102A to turn off, e.g., open, etc., the switch. Thesignal is used to open the switch to facilitate removing a supply ofpower from the power supply to the camera of the wearable device 102A toprevent the camera C1 from capturing images of an item.

Similarly, a camera of the wearable device 102B is turned off or on bydetermining whether the camera is located on a dorsal side or a ventralside of the right arm of the user 302.

In one embodiment, instead of using a camera of a wearable device todetermine whether a camera of the other wearable device, e.g., wearabledevice worn on the other arm of the user 302, etc., is on a dorsal or aventral side of a wrist of the user 302, another camera, e.g., a cameraof the game console 106, the camera 402 (FIG. 4), a camera of the HMD510 (FIG. 5), a camera of a television, etc., is used to capture imagedata of position and orientation of a camera of a wearable device withrespect to a wrist of the user 302.

In an embodiment, inertial sensors of a wearable device worn on an armof a user are used to turn off a camera of the wearable device when thewearable device is at a pre-determined position and/or pre-determinedorientation. In this example, another camera, e.g., a camera of the gameconsole 106, the camera 402 (FIG. 4), a camera of the HMD 510 (FIG. 5),a camera of a television, etc., is used to capture image data of aposition and orientation of another wearable device worn on another armof the user.

In one embodiment, a power supply and a communication device of awearable device are located within, e.g., embedded within, locatedinside, attached to, etc., the wearable device.

In an embodiment, a power supply of a wearable device stores andprovides power to electrical components, e.g., a communication device, alight emitter, a camera, etc., of the wearable device.

FIG. 7A is a diagram of an embodiment of a wearable device 116 thatincludes a number of markers, e.g., a pattern code, or reflectors, orretroreflectors, or light emitting diodes, or quick response (QR) codes,or a combination thereof, etc. In an embodiment, the wearable device 116is made of a reflective fabric or metal. The wearable device 116 is anexample of the wearable device 102A or 102B (FIG. 1A). The markersindicate a position of the wearable device 116 to a camera, e.g., acamera of the game console 106, the camera 402 (FIG. 4), a camera of theHMD 510 (FIG. 5), a camera of a television, a camera of the otherwearable device, which is worn on a wrist other than a wrist on whichthe wearable device 116 is worn, etc. Image data of the markers isgenerated by a camera and the image data is provided to the gameprocessor of the game console or to the processor of the HMD todetermine a position and orientation of an arm of the user 302 on whichthe wearable device 116 is worn with respect to a reference point of thexyz co-ordinate system. The reference point is located on the camerathat captures the image data of the markers.

Another wearable device 118 includes a number of inertial sensors, e.g.,IS1, IS2, IS3, etc., that sense movement of the wearable device 118.Examples of inertial sensors include a magnetometer, an accelerometer, agyroscope, etc. The wearable device 118 is an example of the wearabledevice 102A or 102B (FIG. 1A). The inertial sensors generate signalsindicating movement, e.g., acceleration, orientation with respect to thex, y, and z axis, position with respect to the x, y, and z axis, etc.,of the wearable device 118 and provide the signals to the processor ofthe HMD or of the game console. The processor of the HMD or of the gameconsole uses the position and orientation to identify data regarding astate of a virtual object.

In some embodiments, a wearable device includes markers, and/or cameras,and/or emitters, and/or inertial sensors.

FIG. 7B is a diagram to illustrate that a position of a marker M3 on awearable device 702 is used to determine an orientation of the wearabledevice 702 with respect to an orientation of the wearable device 116.The game processor of the game console 106 determines from image datapositions (y1, z1) and (y2, z2) of the marker M3 with respect to anorigin of the xyz co-ordinate system. The origin of the xyz co-ordinatesystem is located at a camera that captures the image data of the markerM3. A position and orientation of the wearable device 116 does notchange substantially or does not change with respect to a wrist on whichthe wearable device is worn.

The game processor of the game console 106 identifies the orientation ofthe wearable device 702 with respect to the wearable device 116 from acorrespondence, e.g., link, relationship, mapping, etc., between theorientation of the wearable device 702 and a position of the wearabledevice 702. For example, when the position of the wearable device 702 is(y1, z1) with respect to the origin of the xyz co-ordinate system, thegame processor identifies that an orientation of the wearable device 702forms an angle A1 with respect to the z-axis. As another example, whenthe position of the wearable device 702 is (y2, z2) with respect to theorigin of the xyz co-ordinate system, the game processor identifies thatan orientation of the wearable device 702 forms an angle A2 with respectto the z-axis.

In one embodiment, an orientation of a wearable device is the same as anorientation of an arm on which the wearable device is worn. For example,the game processor identifies an orientation of an arm as an orientationof a wearable device that is worn on the arm.

FIG. 8 is a diagram of embodiments of multiple wearable devices 120A,120B, and 120C to illustrate use of fiber optic cables and lightemitters in wearable devices. Each wearable device 120A, 120B, and 120Cis an example of the wearable device 102A or 102B (FIG. 1A). Thewearable device 120A includes a fiber optic cable 810 in a circularpattern to generate diffused fiber optics. For example, the fiber opticcable 810 is opened or torn or cut at various places, e.g., intervals,etc., to generate multiple openings, e.g., an opening O1, an opening O2,an opening O3, etc., to further diffuse light generated by a lightemitter attached to, e.g., soldered to, glued to, etc., an end of thefiber optic cable. Each opening acts as a marker for a camera thatcaptures image data of the wearable device 120A.

The wearable device 120B includes a fiber optic cable 812 in astar-shaped pattern to generate diffused fiber optics. At vertices ofthe star-shaped pattern, one or more openings, e.g., an opening O4, anopening O5, an opening O6, etc., are formed by tearing or cutting thefiber optic cable 812. Light that is emitted by a light emitter travelsthrough the fiber optical cable and is emitted at each opening of thefiber optical cable 812. The light emitter is attached, e.g., glued to,soldered to, etc., to an end of the fiber optic cable 812. Each openingof the fiber optic cable 812 acts as a marker for facilitating a captureof image data of the wearable device 120B.

In an embodiment, a wearable device includes a fiber optic cableembedded therein. The fiber optic cable is made of a medium, e.g., atransparent medium, a translucent medium, etc., which transfers lightthat is emitted by a light source that is attached to the fiber opticcable. When light passes through the fiber optic cable, the cableilluminates and the illumination is captured by a camera, e.g., a cameraof another wearable device, a camera of an HMD, a camera of a gameconsole, an independently-located camera, camera of a television, etc.

The wearable device 120C includes light emitters, e.g., a light emitterLE1, a light emitter LE2, a light emitter LE3, etc., to generate lightthat is emitted at various points, e.g., intervals, etc., along thewearable device 120C. Each light emitter acts as a marker to facilitatecapture of image data by a camera. The light emitters are embeddedwithin an outside surface of the wearable device 120C. The outsidesurface of the wearable device 120C is not adjacent to a wrist of theuser 302 when the wearable device 120C is worn by the user 302.

In one embodiment, a wearable device includes any number of openings foremission of light or any number of light emitters.

In one embodiment, the fiber optic cable 810 or the fiber optic cable812 has another shape, e.g. polygonal, oval, curved, etc.

It should be noted that in one embodiment, each wearable device 120A,120B, and 120C is made of a transparent material, e.g., a transparentplastic, a transparent flexible material, etc., or a semi-transparentmaterial to facilitate light emitted by a light emitter within thewearable device to emit light.

In an embodiment, a first pattern formed by light emitters of a firstwearable device or formed by openings in a fiber optic cable locatedwithin the first wearable device is different than a second patternformed by light emitters of a second wearable device or formed byopenings in a fiber optic cable located within the second wearabledevice. For example, the first pattern is a star-shaped pattern and thesecond pattern is a circular-shaped pattern. The first wearable deviceis worn on the left arm of a user and the second wearable pattern isworn on the right arm of the user. The difference in the patternsfacilitates a game processor of a game console to distinguish positionsand/or orientations of the left hand from positions and/or orientationsof the right hand. For example, in image data of an image that includesboth the patterns, the game processor identifies a position of the lefthand as being a position of the star-shaped pattern and identifies aposition of the right hand as being a position of the circular-shapedpattern.

In one embodiment, a first color of light emitted by light emitters of afirst wearable device or emitted through openings in a fiber optic cableof the first wearable device is different than a second color of lightemitted by light emitters of a second wearable device or of light thatis emitted through openings in a fiber optic cable of the secondwearable device. For example, the first color is green and the secondcolor is red. The first wearable device is worn on the left arm of auser and the second wearable pattern is worn on the right arm of theuser. The difference in the colors facilitates a game processor of agame console to distinguish positions and/or orientations of the lefthand from positions and/or orientations of the right hand. For example,in image data of an image that includes both the colors, the gameprocessor identifies a position of the left hand as being a position ofthe green color and identifies a position of the right hand as being aposition of the red color.

In one embodiment, a wearable device that is worn on one hand of theuser includes one or more light emitters that emit light of a firstwavelength, e.g., infrared light, etc., and another wearable device thatis worn on another hand of the user includes one or more light emittersthat emit light of a second wavelength, e.g., visible light, etc.

FIG. 9 is a diagram of an embodiment of a system for illustrating anumber of cameras, e.g., a camera 910, etc., within the game console 106to determine a relative position and/or orientation of hands of the userwith respect to each other. The camera 910 is integrated within, e.g.,attached to, fitted within, etc., a compartment within the game console106 so that a lens of the camera 910 has a view of the real-worldenvironment in front of the game console 106. The camera 910 of the gameconsole 106 generates image data of an item, e.g., the wearable devices102A and 102B, fingers of hands of the user 302, hands of the user 302,palms of the user 302, etc., and the image data is provided by thecamera 910 to the game processor of the game console 106. The gameprocessor of the game console 106 determines a position and orientationof an item that is displayed in the image data with respect to areference co-ordinate of the xyz co-ordinate system and he referenceco-ordinate is located a location of the camera 910. The game processorfurther identifies data regarding a state, e.g., position, orientation,texture, color, shape, etc., of a virtual object from the position andorientation of an item, and provides data regarding the state to the HMD310 for display on a display screen of the HMD 310.

The data regarding a state of a virtual object is sent via a medium,e.g., a wireless medium, a wired medium, etc., from the game processorof the game console 106 via a communication device of the game console106 and a communication device of the HMD 310 to the processor of theHMD 310. The processor of the HMD 310 renders the data regarding a stateof a virtual object to display the virtual object on a display screen ofthe HMD 310.

In an embodiment, a position and orientation of an item that isdetermined from image data captured using the camera 910 of the gameconsole 106 is used to confirm or deny an accuracy of a position andorientation of an item that is determined from image data captured usinga camera of a wearable device in a manner similar to that describedabove in which image data captured using the camera 512 (FIG. 5) of theHMD 510 (FIG. 5) is used to confirm or deny an accuracy of a positionand orientation of the item determined from image data captured usingthe camera of the wearable device. Upon confirming the accuracy, thegame processor of the game console 106 identifies from the positionand/or orientation of an item determined from image data captured usingthe camera 910 of the game console 106 or using a camera of a wearabledevice data regarding a state of a virtual object, e.g., a virtualfootball, a virtual vehicle, a virtual weapon, a virtual tree, etc., tobe displayed on a display screen of the HMD 310. The game processorsends via a communication device of the game console 106 and acommunication device of the HMD 310, data regarding the state of avirtual object to the HMD 310 for display.

Upon determining that a position and/or orientation of the itemdetermined from the image data captured using the camera of the wearabledevice 102A is not accurate, the game processor of the game console 106waits until the position and/or orientation is determined to be accuratefrom additional image data that is captured using a camera of a wearabledevice and from additional image data that is captured using the camera910 of the game console 106.

In one embodiment, upon determining that a position and/or orientationof the item determined from the image data captured using the camera ofthe wearable device 102A is not accurate, instead of using a positionand/or orientation of an item determined from image data captured usinga camera of a wearable device, the game processor of the game console106 uses a position and/or orientation of an item determined from imagedata captured using the camera 910 of the game console 106 to identifydata regarding a state of a virtual object. The data regardingidentified state is provided to the HMD 310 to display a virtual objecthaving the state on the HMD 310.

It should be noted that image data generated by the camera 910 is from areference point, e.g., origin (0, 0, 0), etc., of the xyz co-ordinatesystem, and the reference point is located at a point on the camera 910of the game console 106, e.g., a point on a lens of the camera 910, etc.

FIG. 10 is a diagram illustrating various gestures performed by the user302 while wearing the wearable devices 102A and 102B. The gestures arethose performed by a traffic cop. For example, one gesture 1002performed using both hands of the user 302 indicates that virtualtraffic, e.g., cars, trucks, rickshaws, bicycles, motorbikes, etc.,travel in a direction pointed to by the hands of the user 302. Thevirtual traffic is displayed on a display screen of the HMD 310. Asanother example, another gesture 1004 also performed using both hands ofthe user 302 indicates that the user 302 is about to draw a virtual gun,which is displayed on a display screen of the HMD 310. As yet anotherexample, a gesture 1006 performed using both hands of the user 302indicates a block motion during a kung fu virtual game, which isdisplayed on a display screen of the HMD 310.

FIG. 11 is a diagram to illustrate two users wearing HMDs 112 andwearable devices 102A and 102B to play games with each other. Forexample, a virtual hand of the user 302 on the left is shown within agame in which a virtual hand of a user 1102 on the right is shown. Asanother example, a virtual head of the user 302 on the left is shownwithin a game in which a virtual head of the user 1102 on the right isshown.

In one embodiment, the user 302 views an avatar of the user 1102 in avirtual environment that is displayed on a display screen of the HMD 310worn by the user 302. Moreover, the user 1102 views an avatar of theuser 302 in the same virtual environment that is displayed on a displayscreen of the HMD 310 worn by the user 1102.

In an embodiment, the users 302 and 1102 are located at the samegeographic location or are located remote from each other. For example,the user 310 is interacting with an avatar of the user 1102 while theuser 310 is located at his house in Texas and the user 1102 isinteracting with an avatar of the user 310 while the user 1102 islocated at his house in California. As another example, both the users310 and 1102 are located in a room and interacting with each other via avirtual environment that is displayed on HMDs 310 worn by the users 302and 1102.

FIG. 12 is a diagram of an embodiment of a system in which a camera 1210of the television 108 is used to determine a position and/or orientationof an item with respect to the xyz co-ordinate system. The camera 1210of the television 108 generates image data of the head of the user andof an item, e.g., the wearable devices 102A and 102B worn by the user302, hands of the user 302, fingers of the user 302, etc. The image datais sent from the camera 1210 to the game processor of the game console106 using a wireless or a wired communication protocol. The gameprocessor of the game console 106 determines a position and/or anorientation of the head of the user 302 with respect to a referencepoint of the xyz co-ordinate system and determines a position and/ororientation of an item associated with the user 302 with respect to thereference point.

The position and/or orientation of an item is used by the game processorof the game console 106 to identify data regarding a state of a virtualobject to be displayed on the HMD 310. It should be noted that a virtualobject corresponds to a position and/or orientation of an item. Forexample, a game memory device of the game console 106 stores acorrespondence, e.g., an association, a mapping, a link, etc., between avirtual object and an item. The game processor of the game console 106accesses the correspondence between the virtual object and the item, andis programmed to affect a state of the virtual object based on aposition and/or orientation of the item, e.g., wearable device, fingers,hand, palm, foot, etc.

It should be noted that image data generated by the camera 1210 is froma reference point, e.g., origin (0, 0, 0), etc., of the xyz co-ordinatesystem, and the reference point is located at a point on the television108, e.g., a point on a lens of the camera 1210, etc.

In an embodiment, a position and orientation of an item that isdetermined from image data captured using the camera 1210 is used toconfirm or deny an accuracy of a position and orientation of an itemthat is determined from image data captured using a camera of a wearabledevice in a manner similar to that described above in which image datacaptured using the camera 512 (FIG. 5) of the HMD 510 (FIG. 5) is usedto confirm or deny an accuracy of a position and orientation of the itemdetermined from image data captured using the camera of the wearabledevice. Upon confirming the accuracy, the game processor of the gameconsole 106 identifies from the position and/or orientation of an itemdetermined from image data captured using the camera 1210 or using acamera of a wearable device data regarding a state of a virtual objectto be displayed on a display screen of the HMD 310 or on a displayscreen of the television 108. The game processor sends via acommunication device of the game console 106 and a communication deviceof the HMD 310, data regarding the state of a virtual object to the HMD310 for display. In one embodiment, the game processor sends via acommunication device of the game console 106 and a communication deviceof the television 108, data regarding a state of a virtual object to aprocessor of the television 108 for display.

Upon determining that a position and/or orientation of the itemdetermined from the image data captured using the camera of the wearabledevice 102A is not accurate, the game processor of the game console 106waits until the position and/or orientation is determined to be accuratefrom additional image data that is captured using a camera of a wearabledevice and from additional image data that is captured using the camera1210.

In one embodiment, upon determining that a position and/or orientationof the item determined from the image data captured using the camera ofthe wearable device 102A is not accurate, instead of using a positionand/or orientation of an item determined from image data captured usinga camera of a wearable device, the game processor of the game console106 uses a position and/or orientation of an item determined from imagedata captured using the camera 1210 to identify data regarding a stateof a virtual object. The data regarding identified state is provided tothe HMD 310 or to the television 108 to display a virtual object havingthe state.

FIG. 13 is a diagram of an embodiment of a system in which ankle devices122A and 122B are worn around an ankle of the user. The ankle device122A is an example of the wearable device 112A and the ankle device 122Bis an example of the wearable device 112B. For example, the ankle device122A has the same shape and performs the same function as that of thewearable device 112A except that the ankle device 112A has a biggerdiameter than that of the wearable device 112A to fit an ankle of theuser 302 instead of a wrist of the user 302. As another example, theankle device 122B has the same shape and performs the same function asthat of the wearable device 112B except that the ankle device 112B has abigger diameter than that of the wearable device 112B to fit an ankle ofthe user 302 instead of a wrist of the user 302.

The ankle device 122A includes one or more cameras that generate imagedata from a view of the right leg of the user 302. The image data isused by the game processor of the game console 106 to calculate arelative position and/or relative orientation of the right leg withrespect to the left leg and/or relative positions and/or relativeorientations of fingers of the right foot with respect to fingers of theleft foot and/or relative position and/or relative orientation of theright foot with respect to the left foot. Moreover, the ankle device122B one or more cameras that generate image data from a view of theleft leg of the user 302. The image data is used by the game processorof the game console 106 to calculate a relative position and/or relativeorientation of the left leg with respect to the right leg and/orrelative positions and/or relative orientations of fingers of the leftfoot with respect to fingers of the right foot and/or relative positionand/or relative orientation of the left foot with respect to the rightfoot.

It should be noted that image data captured using a camera of the ankledevice 122A is used to determine a position and/or orientation of anitem, e.g., the right leg, fingers of the right foot, etc., from areference point, e.g., an origin (0, 0, 0), etc., of the xyz co-ordinatesystem and the reference point is on the camera. Similarly, image datacaptured using a camera of the ankle device 122B is used to determine aposition and/or orientation of an item, e.g., the right leg, fingers ofthe right foot, etc., from a reference point, e.g., an origin (0, 0, 0),etc., of the xyz co-ordinate system and the reference point is on thecamera.

FIG. 14 is a diagram of an embodiment of a system in which the user iswearing wearable devices 102A and 102B around his/her wrist and iswearing ankle devices 122A and 122B around his/her ankles. The wearabledevices 102A and 102B communicate with the HMD 310 to provide image datato the HMD 310 via a wired or a wireless medium and/or communicate withthe game console 106 to provide image data to the HMD game console 310via a wired or a wireless medium. The ankle devices 122A and 122Bcommunicate with the HMD 310 to provide image data to the HMD 310 via awired or a wireless medium and/or communicate with the game console 106to provide image data to the HMD game console 310 via a wired or awireless medium.

FIG. 15 is a diagram of an embodiment of a system in which the user isusing a pad device 1502, e.g., a mat, a surface, a board, a woodenblock, etc., with the wearable devices 102A and 102B. The pad device1502 provides a colored background, e.g., a white background, a greenbackground, a blue background, etc., against which the wearable devices102A and 102B are visible to the camera 512 of the HMD 510. For example,the pad device 1502 is of a color and the wearable devices 102A and 102Bhave a color that stands out against the color of the pad device 1502.

The camera 512 of the HMD 510 captures image data of the wearabledevices 102A and 102B, and of the pad device 1502. The image data iscommunicated from the wearable devices 102A and 102B to the game console106 to determine positions and orientations of the wearable devices 102Aand 102B with respect a reference co-ordinate of the xyz co-ordinatesystem. The reference co-ordinate is located at a lens of the camera512.

In some embodiments, the wearable device 102A is of a different colorthan that of the wearable device 102B.

In one embodiment, the pad device 1502 excludes electronics, e.g., adisplay screen, a processor, a sensor, a camera, etc.

In an embodiment, the wearable device 102A is of a different color ordifferent pattern than the wearable device 102B and the pad 1502. Forexample, the wearable device 102A is of a yellow color or yellow-coloredpattern, the wearable device 102B is of a purple color or purple-coloredpattern, and the pad device 1502 is of a green color or green-coloredpattern. Image data generated by the camera 512 facilitates a gameprocessor of the game console 106 in distinguishing the wearable device102A from the wearable device 102B. The camera 512 is able to generateimage data that facilitates the game processor in distinguishing theyellow color from the purple color when both the colors are placedagainst a background color of green of the pad device 1502. The gameprocessor identifies from a game memory device that the yellow color inimage data corresponds, e.g., maps, associates, etc., to the left handof the user 302, the purple color in image data corresponds to the righthand of the user 302. The game processor determines that movement of thewearable device 102A provides movement of the left hand and movement ofthe wearable device 102B provides movement of the right hand.

In an embodiment, the pad 1502 is passive, e.g., excludes anyelectronics, e.g., sensors, emitters, battery, cameras, etc.

In one embodiment, instead of a passive pad, an active pad, e.g., atablet, a phablet, etc., is used to display a colored screen to providea background against the wearable devices 102A and 102B.

FIG. 16 is a diagram of an embodiment of a system in which the paddevice 1502 is overlaid on a surface, e.g., a desk, a table, a chair,etc., that is supported on a floor. The camera 512 of the HMD 510 or ofthe game console 106 views the wearable devices 102A and 102B againstthe pad device 124 to generate image data. The image data is used todetermine a relative position and/or a relative orientation of the lefthand of the user with respect to the right hand of the user 302.

FIG. 17 is a block diagram of an embodiment of a wearable device 126,e.g., a wearable device, an ankle device, etc. The wearable device 126is an example of the wearable device 102A (FIG. 1A), or the wearabledevice 102B (FIG. 1A), or the wearable device 103A (FIG. 2A), or thewearable device 103B (FIG. 2A), or the wearable device 104A (FIG. 3), orthe wearable device 104B (FIG. 3), or the wearable device 116 (FIG. 7),or the wearable device 118 (FIG. 7), or the wearable device 120A (FIG.8), or the wearable device 120B (FIG. 8), or the wearable device 120C(FIG. 8), or the wearable device 122A (FIG. 13), or the wearable device122B (FIG. 13). The wearable device 126 includes a number of cameras anda communication device 1712. Examples of a communication device includesa communication device that uses a wired communication protocol or awireless communication protocol to communicate information with anothercommunication device. For example, the communication device 1712 usesthe wireless communication protocol, e.g., a Wi-Fi protocol, Bluetooth,etc., to communicate with another communication device. As anotherexample, the communication device 1712 uses a wired communicationprotocol, e.g., a serial transfer protocol, a parallel transferprotocol, a universal serial bus (USB) protocol, etc., to communicatewith another communication device.

One or more cameras of the wearable device 126 that are worn on theright arm of the user 302 capture image data, e.g., one or more images,etc., of an item, e.g., a wearable device that is worn on the left armof the user 302, fingers of the left hand of the user 302, the left handof the user 302, finger joints of fingers of the left hand, etc., togenerate image data. For example, a first camera of the wearable device126 captures images of the left hand when the left hand is at a firstposition and/or a first orientation and a second camera of the wearabledevice 126 captures images of the left hand when the left hand is at asecond position and/or second orientation. A stream, e.g., data packets,digitally encoded coherent signals, etc., of the image data of an itemis communicated from the communication device 1712 using a wired or awireless communication protocol to a communication device 1718 of thegame console 106. For example, the image data is embedded as a payloadinto one or more packets, and a header is included within a packet, tocommunicate the image data in the form of packetized signals. As anotherexample, the communication device 1712 applies a wireless communicationprotocol to generate wireless signals, which are a stream of image data.As yet another example, the communication device 1712 applies a wiredcommunication protocol to generate wired signals, which are a stream ofimage data.

Similarly, a communication device of another wearable device, e.g., awearable device other than the wearable device 126, etc., that is wornon the left arm of the user 302 generates and sends a stream of imagedata, e.g., wireless signals, wired signals, etc., to the communicationdevice 1718. The stream of image data that is generated by the otherwearable device includes image data of the wearable device 126 and/or offingers of the right hand and/or of the right hand and/or of fingerjoints of the right hand and/or of a wearable device that is worn on theright arm of the user 302.

The communication device 1718 of the game console 106 receives thestreams of image data from the communication device 1712 and thecommunication device of the other wearable device worn on the left armof the user, and applies a wireless communication protocol to extractthe image data from the wireless signals or applies a wiredcommunication protocol to extract the image data from the wiredcommunication signals. For example, the communication device 1718 of thegame console 106 rearranges packets of the image data in an order inwhich the image data is generated from a camera of the wearable device126.

The communication device 1718 provides the image data that includespositions and orientations of items, e.g., the wearable devices worn onthe right and left arms, and/or the right and left hands, and/or fingersof the right and left hands, and/or finger joints of the right and lefthands of the user 302, etc., to a game processor 1710 of the gameconsole 106. The game processor 1710 determines, e.g., identifies, etc.,a position and orientation of an item, e.g., the other wearable deviceworn on the right arm of the user 302 and/or of the right hand and/or offingers of the right hand and/or finger joints of one or more fingers ofthe right hand from the image data of the wearable device 126 that isworn on the left arm of the user 302, etc. For example, the gameprocessor 1710 extracts a position, e.g., (x, y, z) co-ordinate, etc.,of the wearable device 126 and/or a finger of the right hand and/or afinger joint of the right hand and/or the right hand from image data ofa portion of the right arm. The image data includes a position andorientation of a portion of the right arm from a reference point, e.g.,an origin (0, 0, 0), etc., of an xyz co-ordinate system. The referencepoint is located at a camera that captures image data.

The game processor 1710 identifies from image data, a position, e.g., an(x, y, z) co-ordinate, etc., of an item with respect to the xyzco-ordinate system of the real-world environment. Examples of an iteminclude a finger joint of a finger, a finger of an arm, a wearabledevice, a hand, a foot, a portion of a leg, a palm, a portion of a palm,etc. As an example, a position of an item from a reference co-ordinate,e.g., origin (0, 0, 0), etc., of the xyz co-ordinate system includes adistance of a point on the item, shown in the image data, along thex-axis of the xyz co-ordinate system from the reference co-ordinate, adistance of the point along the y-axis of the xyz co-ordinate systemfrom the reference co-ordinate, and a distance of the point along thez-axis of the xyz co-ordinate system from the reference co-ordinate. Itshould be noted that the xyz co-ordinate system is located at a camerathat captures image data of the item. For example, when the camera ofthe wearable device 126 captures image data of a hand of the user 302, areference point, e.g., origin, etc., is located at the camera, e.g., alens of the camera, etc., and a position of the hand is identified withrespect to the reference point.

Similarly, the game processor 1710 identifies from image data anorientation of an item with respect to the xyz co-ordinate system of thereal-world environment. As an example, an orientation of an item in animage includes an angle formed by an axis of the item in the image withrespect to the x-axis of the xyz co-ordinate system, an angle formed bythe axis of the item in the image with respect to the y-axis of the xyzco-ordinate system, and an angle formed by the axis of the item in theimage with respect to the z-axis of the xyz co-ordinate system.

In one embodiment, an axis of an item extends along a length of theitem. For example, an axis of a hand extends along a length of the handand an axis of a foot extends along a length of the foot.

In an embodiment, an item shown in an image is referred to herein as avirtual object.

The game processor 1710 further identifies from a game memory device1726 data regarding a state, e.g., position, color, texture, shade,visual effect, audio effect, sound, outline, boundary, shape, etc., of avirtual object to be displayed in a virtual environment corresponding tothe position of an item. For example, the game processor 1710 identifiesfrom the memory device 1726 a mapping between a position of an item anddata regarding a state of a virtual object. As another example, the gameprocessor 1710 determines, from the memory device 1726, a mappingbetween a position of a hand of the user 302 and a position of a virtualhand that represents the hand of the user 302. To illustrate, when theposition of a finger of the user 302 is at a distance (x, y, z) from areference point of the xyz co-ordinate system, a position of a virtualobject is at a distance (X, Y, Z) from a reference point of an XYZco-ordinate system. In this example, the distance (X, Y, Z) in an imageis proportional in each of X, Y, and Z dimension to the distance (x, y,z) in the real-world environment.

It should be noted that in one embodiment, the XYZ co-ordinate system isa virtual co-ordinate system that is used to determine a position andorientation of a virtual object. The xyz co-ordinate system is aco-ordinate system that is used to determine a position and orientationof an item in the real-world environment.

It should be noted that a virtual object whose state is changed based ona position of an item is programmed in the game processor 1710 as beingassociated with the item. For example, it is programmed in the gameprocessor 1710 that a finger of the user 302 is associated with avirtual trigger to allow the finger to control the virtual trigger. Asanother example, it is programmed in the game processor 1710 that aright hand or a left hand of the user 302 is associated with a virtualball to facilitate the user 302 to change a position of a virtual ballby using his/her hand.

Similarly, the game processor 1710 further identifies from the gamememory device 1726 data regarding a state of a virtual object to bedisplayed in a virtual environment corresponding to the orientation ofan item that corresponds to the virtual object. For example, the gameprocessor 1710 identifies from the memory device 1726 a mapping betweenan orientation of an item and a state of a virtual object. As anotherexample, the game processor 1710 determines from the memory deice 1726 amapping between an orientation of a hand of the user 302 and anorientation of a virtual hand that represents the hand of the user 302.As yet another example, when an orientation of a finger of the user 302forms a first angle with respect to the x-axis, a second angle withrespect to the y-axis, and a third angle with respect to the z-axis, thegame processor 1710 identifies from the memory device 1726, anorientation of a virtual object as forming a fourth angle with respectto the X-axis, an orientation of the virtual object as forming a fifthangle with respect to the Y-axis, and an orientation of the virtualobject as forming a sixth angle with respect to the Z-axis. It should benoted that the fourth angle is proportional to the first angle, thefifth angle is proportional to the second angle, and the sixth angle isproportional to the third angle.

The game processor 1710 provides identified data regarding a state of avirtual object to the communication device 1718. The communicationdevice 1718 applies a wired communication protocol or a wirelesscommunication protocol to the identified data regarding a state of avirtual object to generate and send wired or wireless signals.

A communication device 1714 of an HMD 1720 receives the wired signalsfrom the communication device 1718 and applies a wired communicationprotocol to the wired signals to extract identified data regarding astate of a virtual object from the wired signals. Similarly, thecommunication device 1714 of the HMD 1720 applies a wirelesscommunication protocol to the wireless signal to extract identified dataregarding a state of a virtual object from the wireless signals. The HMD1720 is an example of the HMD 310 or the HMD 510.

The communication device 1714 provides data regarding a state that isextracted to an audio/video (A/V) separator. The A/V separator separatesaudio data from image data, both of which are included within the dataregarding a state, sends the image data to the processor 1716 and sendsthe audio data to a synchronizer. The synchronizer synchronizes aplayback of sound with that of a display of a virtual object. Forexample, the synchronizer plays sound at the same time as a virtualobject is displayed at a position and/or an orientation and/or as havinga color and/or as having a shape and/or as having a texture. Thesynchronizer sends the synchronized audio data to a digital to analogconverter that converts the audio data from a digital format into ananalog format. The analog audio data is amplified by an amplifier. Theamplified analog audio data is converted into sound by one or morespeakers.

The processor 1716 of the HMD 1720 renders image data regarding a stateof a virtual object to display the state of the virtual object on adisplay screen 1122. For example, the processor 1720 displays a virtualobject as having a color, a shape, a texture, a position, anorientation, etc., in a virtual environment. Examples of a displayscreen include a liquid crystal display (LCD) screen, an LED displayscreen, a plasma display screen, etc.

Examples of a memory device include a hard drive, a network attachedstorage (NAS), a read-only memory (ROM), a random-access memory (RAM), acompact disc-ROMs (CD-ROMs), a CD-recordable (CD-R), a CD-rewritable(CD-RW), a magnetic tape, and other optical or non-optical data storagedevice.

In one embodiment, the game processor 1710 of the game console 106receives image data that is captured by a camera of a television or acamera of an HMD or an independently-located camera, e.g., the camera402 (FIG. 4), etc., and determines from the image data a position and/ororientation of an item in the real-world environment.

In an embodiment, a camera of a wearable device that is worn on a rightwrist of the user captures image data representing a portion of a leftarm of the user. The image data is used by the game processor todetermine a position and orientation of the portion of the left arm ofthe user. The position and/or orientation are used to identify a stateof a portion of a virtual left arm of the user. The portion of thevirtual left arm is displayed on a display screen of an HMD. As anexample, as the user moves his/her left arm to be within a field-of-viewof a camera worn on the right wrist of the user, the portion of thevirtual left arm enters a virtual environment displayed on the displayscreen of the HMD. Similarly, in this embodiment, a camera of a wearabledevice that is worn on a left wrist of the user captures image dataregarding a portion of a right arm of the user. The image data capturedby the wearable device worn on the left wrist is used by the gameprocessor to determine a position and orientation of the portion of theright arm of the user. The position and/or orientation of the right armof the user are used to identify a state of a portion of a virtual rightarm of the user. The portion of the virtual right arm is displayed onthe display screen of the HMD. For example, as the user moves his/herright arm to be within a field-of-view of a camera worn on the leftwrist of the user, the portion of the virtual right arm enters a virtualenvironment displayed on the display screen of the HMD.

In an embodiment, one or more cameras of the wearable device 126 captureimage data and/or streaming of image data from one communication deviceto another communication device occurs during a session of presenting avirtual environment, e.g., a virtual reality scene, an augmented realityscene, etc., in an HMD. For example, while a user is playing a gameusing an HMD, cameras of the wearable device 126 worn on a left wrist ofthe user and cameras of another wearable device worn on the right wristof the user capture image data. As another example, while a user isinteracting with a virtual environment that is displayed in an HMD,image data that is captured using cameras of the wearable device 126 isstreamed from the communication device 1712 of the wearable device 126to the communication device 1718 of the game console 106

In an embodiment, a session of a virtual environment is presented on adisplay screen of an HMD when the user 302 places the HMD on his/herhead to cover his/her eyes.

In one embodiment, a session of a virtual environment is presented on adisplay screen when user information, e.g., username, user password,etc., assigned to the user 302 is authenticated by an authenticationserver that is connected to a communication device of the game console106. The user 302 provides the user information using an input device,e.g., a game controller, a keypad, a keyboard, a mouse, etc., to thegame processor 1710 of the game console 106. The game processor 1710sends the user information via a network device, e.g., a networkinterface controller, a network interface card, a modem, etc., of thegame console 106 and a computer network, e.g., the Internet, anIntranet, a combination thereof, etc., to the authentication server. Theauthentication server determines whether the user information isauthentic and provides a result of the authentication to the gameprocessor 1710. Upon receiving the indication that the user informationis authentic, the game processor 1710 provides state data of a virtualenvironment via the communication device 1718 and the communicationdevice 1714 to the processor 1716. The processor 1716 renders the statedata to initiate a session of a virtual environment during which thevirtual environment is displayed to the user 302 via the display screen1722 of the HMD 1720.

In one embodiment, multiple communication devices, e.g., communicationdevices 1718 and 1714, communication devices 1712 and 1718, etc., areinterfaced with each other when the communication devices are capable ofcommunicating data, e.g., image data, audio data, etc., with each other.

In an embodiment, the game processor 1710 receives image data used todisplay multiple images to identify changes in positions of fingers of ahand of the user 302. For example, the game processor 1710 identifies afirst position of a finger from a first image and a second position ofthe finger from a second image. The game processor 1710 identifieschanges in positions of the finger as a change from the first positionto the second position. The second position occurs in time after thefirst position occurs. It should be noted that the first and secondimages are captured by the same camera of the wearable device 126 or bymultiple cameras of the wearable device 126.

In one embodiment, the wearable device 126 includes multiple inertialsensors, e.g., gyroscopes, magnetometers, accelerometers, etc., thatgenerate motion data, e.g., orientation of a hand of the user, magneticfields, changes in rotational velocity, acceleration of the hand, etc.The motion data is communicated from the communication device 1712 ofthe wearable device 126 to the communication device 1718 of the gameconsole 106 in a form of a stream, e.g., wireless signals generatedafter applying a wireless communication protocol to the motion data,wired signals generated after applying a wired communication protocol tothe motion data, etc. The communication device 1718 of the game console106 extracts the motion data by applying a wired communication protocolto the wired signals or by applying a wireless communication protocol tothe wireless signals. The game processor 1710 receives the motion datafrom the communication device 1718. The game processor 1710 determinesone or more positions and/or one or more orientations of a hand of theuser from the motion data.

In an embodiment, upon determining that image data captured using acamera of the wearable device 126 does not include image data of a handof the user, the game processor 1710 uses motion data to determine aposition and/or orientation of the hand of the user.

In one embodiment, the game processor 1710 uses both motion data andimage data to determine a position and/or orientation of a hand of theuser. For example, the game processor 1710 generates a statisticalvalue, e.g., an average, etc., of a position determined from motion datagenerated by a wearable device and a position determined from image dataincluding an image of the wearable device to calculate a position of thewearable device. The calculated position is used to identify a state ofa virtual object. As another example, the game processor 1710 generatesa statistical value, e.g., an average, etc., of an orientationdetermined from motion data generated by a wearable device and anorientation determined from image data including an image of thewearable device to calculate an orientation of the wearable device. Thecalculated orientation is used to identify a state of a virtual object.

In one embodiment, a camera of the wearable device 126 detects lightthat is emitted from a light emitter of another wearable device, e.g.,wearable device worn on an arm other than an arm on which the wearabledevice 126 is worn, etc., to generate electrical signal data. Theelectrical signal data is received by the game processor 1710 in amanner similar to that described above and is processed by the gameprocessor 1710 to determine an orientation of an arm of the user onwhich the other wearable device is worn. For example, upon determiningthat the light is of a blue color, the game processor 1710 identifiesbased on a correspondence between the color of light and an orientationof the hand that the hand is oriented so that the first lateral side ofthe hand faces the camera of the wearable device 126. As anotherexample, upon determining that the light is of a green color, the gameprocessor 1710 identifies based on a correspondence between the color oflight and an orientation of the hand that the hand is oriented so thatthe dorsal side of the hand faces the camera of the wearable device 126.A correspondence, e.g., an association, a link, a mapping, etc., betweena side of a hand of the user and a color of light is stored in the gamememory device 1726.

In an embodiment, instead of using a color of light to determine anorientation of a hand of the user, an intensity of light or a shade oflight is used to determine the orientation. For example, the lightemitter LES13 (FIG. 1C) emits light having a greater intensity thanlight emitted by the light emitter LES1 (FIG. 1C). It should be notedthat a color of light and an intensity of the light are examples ofcharacteristics of the light.

In one embodiment, both arms of the user 302 move synchronously orsubstantially synchronously with each other. For example, both arms movein the same direction, e.g., up, down, left, right, etc., orsubstantially in the same direction. When both arms move synchronouslyor substantially synchronously with each other, a reference camera,e.g., a camera of the game console 106, a camera of the HMD 510 (FIG.5), etc., is used to capture an image of hands of the user 302. Theimage data captured using the reference camera is communicated, usingthe wired communication protocol or the wireless communication protocol,from a communication device of the reference camera to the communicationdevice 1718 of the game console 106. The communication device 1718applies the wired communication protocol or the wireless communicationprotocol to extract and provide the image data to the game processor1710. The game processor 1710 receives the image data. Moreover, thegame processor 1710 receives image data from cameras of wearable devicesworn on left and right arms of the user 302. For example, the gameprocessor 1710 receives image data that includes positions andorientations of the right hand from the wearable device 102A (FIG. 1A)and receives image data that includes positions and orientations of theleft hand from the wearable device 102B (FIG. 1A). The game processor1710 determines whether position and/or orientation of both hands of theuser 302 determined from image data captured by the cameras of thewearable devices changes during a pre-determined time period. Forexample, the game processor 1710 determines whether position and/ororientation of the left hand changes in image data captured using acamera of the wearable device 102B and determines whether positionand/or orientation of the right hand changes in image data capturedusing a camera of the wearable device 102A. The game processor 1710further determines whether position and/or orientation of both hands ofthe user 302 determined from image data captured by the reference camerachanges during the pre-determined time period. Upon determining that theposition and/or orientation of both hands determined from image datacaptured by the cameras of the wearable devices does not change duringthe pre-determined time period and position and/or orientation of bothhands of the user 302 determined from image data captured by thereference camera changes during the pre-determined time period, the gameprocessor 1710 determines that the position and/or orientation of bothhands changes with respect to a reference point, e.g., an origin (0, 0,0) of the xyz co-ordinate system, etc., located at a location of thereference camera.

In one embodiment, upon determining that the position and/or orientationof both hands determined from image data captured by the cameras of thewearable devices does not change during the pre-determined time periodand position and/or orientation of both hands of the user 302 determinedfrom image data captured by the reference camera changes during thepre-determined time period, the game processor 1710 determines to applythe position and/or orientation of both hands that are determined fromthe image data captured using the reference camera to further identify astate of a virtual object instead of applying image data captured usingthe cameras of the wearable devices.

In one embodiment, the game processor 1710 determines from image datacaptured using a camera, e.g., a camera of a wearable device, a cameraof a game console, a camera of an HMD, an independently-located camera,etc., whether a difference in two consecutively-determined positions isgreater than a pre-determined threshold. For example, when a hand of theuser keeps moving in and out of a field-of-view of a camera, thedifference in two consecutively-determined positions is greater than thepre-determined threshold. In this example, the hand keeps moving in andout of the field-of-view of the camera when a wearable device thatincludes the camera slides around a wrist on which the wearable deviceis worn and/or when the user moves his/her hand of an arm on which thecamera is located and/or when the user moving his/her other hand and/orwhen an object is creating an obstruction between the camera and theother hand. Upon determining that the difference is greater than thepre-determined threshold, the game processor 1710 interpolates positionsbetween the two consecutively-determined positions. For example, thegame processor 1710 determines a speed of movement of a hand of the user302 from positions of the hand of the user 302 and time period betweenthe positions, and determines from the speed and time passed between oneof the positions and a position occurring between the twoconsecutively-determined positions the position between the twoconsecutively-determined positions. As another example, the gameprocessor 1710 connects the two consecutively-determined positions togenerate positions between the consecutively-determined positions. Theinterpolation creates image stabilization of images of a virtual objectthat is displayed as having positions that correspond to positions ofthe hand that is represented by the virtual object.

In one embodiment, the communication device 1712 of the wearable device126, e.g. a first wearable device, etc., that is worn on a wrist of auser receives wired or wireless signals that include first image datafrom a communication device of a second wearable device that is worn onanother wrist of the user. The first image data is captured by a cameraof the second wearable device. The communication device 1712 alsoreceives second image data captured by a camera of the first wearabledevice and applies a communication protocol, e.g., the wired protocol,the wireless protocol, etc., to the first and second image data togenerate signals and to communicate the signals to the communicationdevice 1718 of the game console 106. The communication device 1712further identifies within the signals that the first image data isreceived from the second wearable device and the second image data isobtained from the first wearable device. The communication device 1718receives the signals from the communication device 1712 and applies thecommunication protocol to extract the first and second image data forprovision to the game processor 1710. The game processor 1710distinguishes between the first and second image data using theidentification of the first image data and the identification of thesecond image data.

FIG. 18A is a diagram of an embodiment of an image 1816 of a virtualenvironment that is displayed on an HMD, e.g., the HMD 310, the HMD 510,etc., to illustrate that one or both hands of the user 302 are used tocontrol a virtual object 1810, e.g., a volleyball, a soccer ball, abasketball, a bowling ball, etc., that is displayed within the image1816. The game processor of the game console 106 associates, e.g.,links, maps, etc., the virtual object 1810 with position and/ororientation of one or both hands of the user 302. For example, when oneor both hands move with respect to the xyz co-ordinate system associatedwith the real-world environment, the game processor of the game console106 moves the virtual object 1810 with respect to the XYZ co-ordinatesystem.

It should be noted that the game processor of the game console 106identifies a position of the virtual object 1810 as being proportionalto a position of one or both hands of the user 302. For example, whenone or both hands move closer to a reference point of an xyz co-ordinatesystem of the real-world environment to be located at a distance (x, y,z) from the reference point, the virtual object 1810 moves closer to areference point of the XYZ co-ordinate system to be located at adistance (X, Y, Z) from the reference point of the XYZ co-ordinatesystem. The distance (X, Y, Z) is proportional to the distance (x, y,z). Similarly, the game processor of the game console 106 identifies anorientation of the virtual object 1810 as being proportional to anorientation of one or both hands of the user 302. For example, when bothhands are at an angle with respect to an x-axis of an xyz co-ordinatesystem, an angle with respect to the y-axis of the xyz co-ordinatesystem, and at an angle with respect to the z-axis of the xyzco-ordinate system, the virtual object 1810 is located at an angle withrespect to the X-axis of the XYZ co-ordinate system, an angle withrespect to the Y-axis of the XYZ co-ordinate system, and at an anglewith respect to the Z-axis of the XYZ co-ordinate system. The angle withrespect to the X-axis of the XYZ co-ordinate system is proportional tothe angle with respect to the x-axis of the xyz co-ordinate system, theangle with respect to the Y-axis of the XYZ co-ordinate system isproportional to the angle with respect to the y-axis of the xyzco-ordinate system, and the angle with respect to the Z-axis of the XYZco-ordinate system is proportional to the angle with respect to thez-axis of the xyz co-ordinate system.

In one embodiment, a position of both hands of the user 302 with respectto a reference point of the xyz co-ordinate system is identified by thegame processor of the game console 106 to be the same as a position of apoint between the two hands that is equidistant from the two hands.Moreover, in this embodiment, an orientation of both hands of the user302 with respect to the xyz co-ordinate system is identified by the gameprocessor of the game console 106 to be the same as an orientation of anaxis that is equidistant between axes of the two hands.

FIG. 18B is a diagram of an embodiment of an image 1818 of a virtualenvironment that is displayed on an HMD, e.g., the HMD 310, the HMD 510,etc., to illustrate that one hand, e.g., right hand, left hand, etc., ofthe user 302 is used to control a virtual object 1812, e.g., a tennisball, a weight-lifting exercise ball, a baseball, a cricket ball, etc.,and another hand of the user 302 is used to control another virtualobject 1814, e.g., a tennis ball, a weight-lifting exercise ball, abaseball, a cricket ball, etc., within an image 1818. The game processorof the game console 106 associates, e.g., links, maps, etc., the virtualobject 1812 with position and/or orientation of the left hand of theuser 302 and associates the virtual object 1814 with the right hand ofthe user 302. For example, when the left hand is moved with respect to areference point of the xyz co-ordinate system, the game processor of thegame console 106 moves the virtual object 1812 with respect to areference point of the XYZ co-ordinate system and when the right hand ismoved with respect to the reference point of the xyz co-ordinate system,the game processor of the game console 106 moves the virtual object 1814with respect to the reference point of the XYZ co-ordinate system. Asanother example, the movement of the left hand of the user 302 does notaffect movement of the virtual object 1814 and the movement of the righthand of the user 302 does not affect movement of the virtual object1812.

FIG. 19 is an isometric view of an HMD 2100, which is an example of theHMD 310 (FIG. 3). The HMD 2100 includes bands 2102 and 2104 that go tothe back of the head of the user 302 when worn by the user 302.Moreover, the HMD 2100 includes earphones 2106A and 2106B, e.g.,speakers, etc., that emanate sound associated with a virtualenvironment, e.g., a game environment, a virtual tour environment, etc.,that is played by execution of a computer program, e.g., a game program,a virtual environment generation program, etc. The HMD 2100 includeslenses 2108A and 2108B that allows the user 302 to view a virtualenvironment that is displayed on a display screen of the HMD 2100. Agroove 2180 rests on a nose of the user 302 to support the HMD 2100 onthe nose.

In some embodiments, an HMD 2100 is worn by the user 302 in a mannersimilar to which sunglasses, glasses, or reading glasses are worn by theuser 302.

FIG. 20 illustrates a system for interactive game play of a video game,in accordance with an embodiment described in the present disclosure.The user 302 is shown wearing the HMD 310. The HMD 310 is worn in amanner similar to glasses, goggles, or a helmet, and is configured todisplay a video game or other content to the user 302. The HMD 310provides an immersive experience to the user by virtue of its provisionof display mechanisms (e.g., optics and display screens) in closeproximity to the user's eyes and the format of content that is deliveredto the HMD 310. In one example, the HMD 310 provides display regions toeach of the user's eyes which occupy large portions or even the entiretyof the field of view of the user 302.

In one embodiment, the HMD 310 is connected to a computer 2202. Theconnection to computer 2202 can be wired or wireless. The computer 2202,in one embodiment, is any general or special purpose computer, includingbut not limited to, a game console, a personal computer, a laptop, atablet, a mobile device, a smart phone, a tablet, a thin client, aset-top box, a media streaming device, a smart television, etc. In someembodiments, the HMD 310 can connect directly to the Internet, which mayallow for cloud gaming without the need for a separate local computer.In one embodiment, the computer 2202 is configured to execute a videogame (and other digital content), and output the video and audio fromthe video game for rendering by the HMD 310. The computer 2202 is alsosometimes referred to herein as a client system, which in one example isa video game console.

The computer 2202 may, in some embodiments, is a local or remotecomputer, and the computer runs emulation software. In a cloud gamingembodiment, the computer 2202 is remote and may be represented by aplurality of computing services that may be virtualized in data centers,where game systems/logic is virtualized and distributed to the user 302over a computer network.

The user 302 operates a hand-held controller 2206 to provide input for avirtual environment. In one example, a camera 2204 is configured tocapture image of the real-world environment in which the user 302 islocated. These captured images are analyzed to determine a location andmovements of the user 302, the HMD 310, and the controller 2206. In oneembodiment, the controller 2206 includes a light (or lights) which aretracked to determine its location and orientation. Additionally, asdescribed in further detail below, in one embodiment, the HMD 310includes one or more lights, which are tracked as markers to determinethe location and orientation of the HMD 310 in substantial real-timeduring a display of a virtual environment.

The camera 2204, in one embodiment, includes one or more microphones tocapture sound from the real-world environment. Sound captured by amicrophone array is processed to identify the location of a soundsource. Sound from an identified location is selectively utilized orprocessed to exclusion of other sounds not from the identified location.Furthermore, in one embodiment, the camera 2204 is configured to includemultiple image capture devices (e.g. stereoscopic pair of cameras), anIR camera, a depth camera, and combinations thereof.

In some embodiments, computer 2202 executes games locally on theprocessing hardware of the computer 2202. The games or content isobtained in any form, such as physical media form (e.g., digital discs,tapes, cards, thumb drives, solid state chips or cards, etc.) or by wayof download from a computer network 2210, e.g., the Internet, anIntranet, a local area network, a wide area network, etc. In anembodiment, the computer 2202 functions as a client in communicationover the computer network 2210 with a cloud gaming provider 2212. Thecloud gaming provider 2212 maintains and executes the video game beingplayed by the user 302. The computer 2202 transmits inputs from the HMD310, the controller 2206, and the camera 2204, to the cloud gamingprovider 2212, which processes the inputs to affect the game state ofthe video game being executed. The output from the executing video game,such as video data, audio data, and haptic feedback data, is transmittedto the computer 2202. The computer 2202 further processes the databefore transmission or directly transmits the data to the relevantdevices. For example, video and audio streams are provided to the HMD310, whereas a vibration feedback command is provided to the controller2206.

In one embodiment, the HMD 310, controller 2206, and camera 2204, arenetworked devices that connect to the computer network 2210 tocommunicate with the cloud gaming provider 2212. For example, thecomputer 2202 may be a local network device, such as a router, that doesnot otherwise perform video game processing, but facilitates passage ofnetwork traffic. The connections to the computer network 2210 by the HMD310, controller 2206, and camera 2204 are wired or wireless. In someembodiments, content executed on the HMD 310 or displayable on a displaydevice 2214, is obtained from any of content sources 2216. Examplecontent sources can include, for instance, internet websites thatprovide downloadable content and/or streaming content. In some examples,the content can include any type of multimedia content, such as movies,games, static/dynamic content, pictures, social media content, socialmedia websites, virtual tour content, cartoon content, etc.

In one embodiment, the user 302 is playing a game on the HMD 310, wheresuch content is immersive 3D interactive content. The content on the HMD310, while the player is playing, is shared to the display device 2214.In one embodiment, the content shared to the display device 2214 allowsother users proximate to the user 302 or remote to watch along with gameplay of the user 302. In still further embodiments, another playerviewing the game play of user 302 on the display device 2214participates interactively with user 302. For example, a user viewingthe game play on the display device 2214 controls characters in the gamescene, provides feedback, provides social interaction, and/or providescomments (via text, via voice, via actions, via gestures, etc.,) whichenables the user who is not wearing the HMD 310 to socially interactwith the user 302.

FIG. 21 illustrates a head-mounted display (HMD) 2300, in accordancewith an embodiment described in the present disclosure. The HMD 2300 isan example of the HMD 510 (FIG. 5). As shown, the HMD 2300 includes aplurality of lights 2302A-H, J and K (e.g., where 2302K and 2302J arelocated toward the rear or backside of the HMD headband). Each of theselights are configured to have specific shapes and/or positions, and areconfigured to have the same or different colors. The lights 2302A,2302B, 2302C, and 2302D are arranged on the front surface of the HMD2300. The lights 2302E and 2302F are arranged on a side surface of theHMD 2300. And the lights 2302G and 2302H are arranged at corners of theHMD 2300, so as to span the front surface and a side surface of the HMD2300. It will be appreciated that the lights are identified in capturedimages of an interactive environment in which a user uses the HMD 2300.

Based on identification and tracking of the lights, the location andorientation of the HMD 2300 in the interactive environment isdetermined. It will further be appreciated that some of the lights areor are not visible depending upon the particular orientation of the HMD2300 relative to an image capture device, e.g., a camera. etc. Also,different portions of lights (e.g. lights 2302G and 2302H) are exposedfor image capture depending upon the orientation of the HMD 2300relative to the image capture device. In some embodiments, inertialsensors are disposed in the HMD 2300, which provide feedback regardingpositioning, without the need for lights. In some embodiments, thelights and inertial sensors work together, to enable mixing andselection of position/motion data.

In one embodiment, the lights are configured to indicate a currentstatus of the HMD 2300 to others users in the real-world environment.For example, some or all of the lights are configured to have a colorarrangement, an intensity arrangement, be configured to blink, have acertain on/off configuration, or other arrangement indicating a currentstatus of the HMD 2300. By way of example, the lights are configured todisplay different configurations during active game play of a video game(generally game play occurring during an active timeline or within ascene of the game) versus other non-active game play aspects of a videogame, such as navigating menu interfaces or configuring game settings(during which the game timeline or scene is inactive or paused).

In an embodiment, the lights are also configured to indicate relativeintensity levels of game play. For example, the intensity of lights, ora rate of blinking, increases when the intensity of game play increases.

The HMD 2300, in one embodiment, additionally includes one or moremicrophones. In the illustrated embodiment, the HMD 2300 includesmicrophones 2304A and 2304B located on the front surface of the HMD2300, and a microphone located on a side surface of the HMD 2300. Byutilizing an array of microphones, sound from each of the microphones isprocessed to determine a location of the sound's source. Thisinformation is utilized in various ways, including exclusion of unwantedsound sources, association of a sound source with a visualidentification, etc.

The HMD 2300 includes one or more image capture devices. In theillustrated embodiment, the HMD 2300 is shown to include image captureddevices 2306A and 2306B. In an embodiment, by utilizing a stereoscopicpair of image capture devices, three-dimensional (3D) images and videoof the real-world environment is captured from the perspective of theHMD 2300. Such video is presented to the user 302 to provide the userwith a “video see-through” ability while wearing the HMD 2300. That is,though the user cannot see through the HMD 2300 in a strict sense, thevideo captured by the image capture devices 2306A and 2306B nonethelessprovides a functional equivalent of being able to see the real-worldenvironment external to the HMD 2300 as if looking through the HMD 2300.

Such video, in one embodiment, is augmented with virtual elements toprovide an augmented reality experience, or is combined or blended withvirtual elements in other ways. Though in the illustrated embodiment,two cameras are shown on the front surface of the HMD 2300, it will beappreciated that there may be any number of externally facing cameras ora single camera can be installed on the HMD 2300, and oriented in anydirection. For example, in another embodiment, there may be camerasmounted on the sides of the HMD 2300 to provide additional panoramicimage capture of the environment.

FIG. 22 illustrates one example of game play using a client system 2402that is capable of rendering the video game content to the HMD 2300 ofthe user 302. In this illustration, a state of a virtual object, e.g.,game content, etc., provided to the HMD 2300 is in a rich interactive3-D space. As discussed above, a state of a virtual object is downloadedto the client system 2402 or is executed in one embodiment by a cloudprocessing system. Cloud gaming service 2212 includes a database ofusers 2404, which are allowed to access particular games 2430, shareexperiences with other friends, post comments, and manage their accountinformation.

The cloud gaming service 2212 stores game data 2406 for specific users,which may be usable during game play, future game play, sharing to asocial media network, or used for storing trophies, awards, status,ranking, etc. Social data 2408 is managed by cloud gaming service 2212.In one embodiment, the social data 2408 is managed by a separate socialmedia network, which is interfaced with cloud gaming service 2212 overthe computer network 2210. Over the computer network 2210, any number ofclient systems 2410 are connected for access to the content andinteraction with other users.

Continuing with the example of FIG. 24, the three-dimensionalinteractive scene viewed in the HMD 2300 includes game play, such as thecharacters illustrated in the 3-D view, or another virtual environment.One character, e.g. P1, etc., is controlled by the user 302 that iswearing the HMD 2300. This example shows a basketball scene between twoplayers, wherein the HMD user 302 is dunking a ball on another characterin the 3-D view. The other character can be an AI (artificialintelligence) character of the game, or can be controlled by anotherplayer or players (Pn). User 302, who is wearing the HMD 2300, is shownmoving about in a space of use, where the HMD 2300 moves around based onthe user's head movements and body positions. A camera 2412 is shownpositioned over a display screen in the room, however, for HMD use, thecamera 2412 can be placed in any location that can capture images of theHMD 2300. As such, the user 302 is shown turned at about 90 degrees fromthe camera 2412 and a display device 2212, as content rendered in theHMD 2300 can be dependent on the direction that the HMD 2300 ispositioned, from the perspective of the camera 2412. Of course, duringHMD use, the user 302 will be moving about, turning his head, looking invarious directions, as is needed to take advantage of the dynamicvirtual scenes rendered by the HMD 2300.

FIG. 23 illustrates a user wearing the HMD 2300, during use, inaccordance with one embodiment. In this example, it is shown that theHMD 2300 is tracked 2502 using image data obtained from captured videoframes by the camera 2412. Additionally, it is shown that the hand-heldcontroller 2206 is also be tracked 2504 using image data obtained fromcaptured video frames by the camera 2412. Also shown is theconfiguration where the HMD 2300 is connected to the computing system2202 via a cable 2510. In one embodiment, the HMD 2300 obtains powerfrom the same cable or can connect to another cable. In still anotherembodiment, the HMD 2300 has a battery that is rechargeable, so as toavoid extra power cords.

With reference to FIG. 24, a diagram is shown illustrating examplecomponents of a HMD 2600, in accordance with an embodiment described inthe present disclosure. The HMD 2600 is an example of the HMD 510 (FIG.5). When the HMD 2600 excludes any cameras, the HMD 2600 is an exampleof the HMD 310 (FIG. 3). It should be understood that more or lesscomponents can be included or excluded from the HMD 2600, depending onthe configuration and functions enabled. The HMD 2600 includes aprocessor 2602 for executing program instructions. A memory 2604 isprovided for storage purposes, and in one embodiment, includes bothvolatile and non-volatile memory. A display 2606 is included whichprovides a visual interface that the user 302 views.

The display 2606 is defined by one single display, or in the form of aseparate display screen for each eye. When two display screens areprovided, it is possible to provide left-eye and right-eye video contentseparately. Separate presentation of video content to each eye, forexample, can provide for better immersive control of three-dimensional(3D) content. As described herein, in one embodiment, the second screenis provided with second screen content of the HMD 2600 by using theoutput for one eye, and then formatting the content for display in a 2Dformat. The one eye, in one embodiment, can be the left-eye video feed,but in other embodiments it can be the right-eye video feed.

A battery 2608 is provided as a power source for the HMD 2600. In otherembodiments, the power source includes an outlet connection to power. Inother embodiments, an outlet connection to power and the battery 2608are provided. A motion detection module 2610 includes any of variouskinds of motion sensitive hardware, such as a magnetometer 2612, anaccelerometer 2614, and a gyroscope 2616.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis (e.g., six-axis)models are able to detect magnitude and direction of the acceleration indifferent directions. The accelerometer is used to sense inclination,vibration, and shock. In one embodiment, three accelerometers are usedto provide the direction of gravity, which gives an absolute referencefor two angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of an HMD. In one embodiment, three magnetometers areused within an HMD, ensuring an absolute reference for the world-spaceyaw angle. In one embodiment, the magnetometer is designed to span theearth magnetic field, which is ±80 microtesla. Magnetometers areaffected by metal, and provide a yaw measurement that is monotonic withactual yaw. The magnetic field is warped due to metal in theenvironment, which causes a warp in the yaw measurement. If necessary,this warp is calibrated using information from other sensors such as thegyroscope or the camera. In one embodiment, accelerometer 2614 is usedtogether with magnetometer 2612 to obtain the inclination and azimuth ofthe HMD 2600.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes provide information about movement across the respective axis(x, y and z) based on inertial sensing. The gyroscopes help in detectingfast rotations. However, the gyroscopes drift overtime without theexistence of an absolute reference. To reduce the drift, the gyroscopesare reset periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 2618 is provided for capturing images and image streams of thereal-world environment. In one embodiment, more than one camera(optionally) is included in the HMD 2600, including a camera that isrear-facing (directed away from the user 302 when the user 302 isviewing the display of the HMD 2600), and a camera that is front-facing(directed towards the user 302 when the user is viewing the display ofthe HMD 2600). Additionally, in an embodiment, a depth camera 2620 isincluded in the HMD 2600 for sensing depth information of objects in thereal-world environment.

The HMD 2600 includes speakers 2622 for providing audio output. Also, inone embodiment, a microphone 2624 is included for capturing audio fromthe real-world environment, including sounds from the ambientenvironment, speech made by the user 302, etc. In an embodiment, the HMD2600 includes tactile feedback module 2626 for providing tactilefeedback to the user 302. In one embodiment, the tactile feedback module2626 is capable of causing movement and/or vibration of the HMD 2600 soas to provide tactile feedback to the user 302.

LEDs 2630 are provided as visual indicators of statuses of the HMD 2600.For example, an LED indicates battery level, power on, etc. A cardreader 2632 is provided to enable the HMD 2600 to read and writeinformation to and from a memory card. A USB interface 2634 is includedas one example of an interface for enabling connection of peripheraldevices, or connection to other devices, such as other portable devices,computers, etc. In various embodiments of the HMD 2600, any of variouskinds of interfaces may be included to enable greater connectivity ofthe HMD 2600.

In an embodiment, a Wi-Fi module 2636 is included for enablingconnection to the computer network via wireless networking technologies.Also, in one embodiment, the HMD 2600 includes a Bluetooth module 2638for enabling wireless connection to other devices. A communications link2640 is included for connection to other devices. In one embodiment, thecommunications link 2640 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 2640utilizes any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 2642 are included to provide an input interfacefor the user 302. Any of various kinds of input interfaces may beincluded, such as buttons, gestures, touchpad, joystick, trackball, etc.In one embodiment, an ultra-sonic communication module 2644 is includedin HMD 2600 for facilitating communication with other devices viaultra-sonic technologies.

In an embodiment, bio-sensors 2646 are included to enable detection ofphysiological data from the user 302. In one embodiment, the bio-sensors2646 include one or more dry electrodes for detecting bio-electricsignals of the user 302 through the user's skin, voice detection, eyeretina detection to identify users/profiles, etc.

The foregoing components of HMD 2600 have been described as merelyexemplary components that may be included in HMD 2600. In variousembodiments described in the present disclosure, the HMD 2600 may or maynot include some of the various aforementioned components. Embodimentsof the HMD 2600 may additionally include other components not presentlydescribed, but known in the art, for purposes of facilitating aspects ofthe present invention as herein described.

It will be appreciated by those skilled in the art that in variousembodiments described in the present disclosure, the aforementionedhandheld device is utilized in conjunction with an interactiveapplication displayed on a display to provide various interactivefunctions. The exemplary embodiments described herein are provided byway of example only, and not by way of limitation.

In one embodiment, clients and/or client devices, as referred to herein,may include head mounted displays (HMDs), terminals, personal computers,game consoles, tablet computers, telephones, set-top boxes, kiosks,wireless devices, digital pads, stand-alone devices, handheld gameplaying devices, and/or the like. Typically, clients are configured toreceive encoded video streams, decode the video streams, and present theresulting video to a user, e.g., a player of a game. The processes ofreceiving encoded video streams and/or decoding the video streamstypically includes storing individual video frames in a receive bufferof the client. The video streams may be presented to the user on adisplay integral to client or on a separate device such as a monitor ortelevision.

Clients are optionally configured to support more than one game player.For example, a game console may be configured to support two, three,four or more simultaneous players (e.g., P1, P2, . . . Pn). Each ofthese players receives or shares a video stream, or a single videostream may include regions of a frame generated specifically for eachplayer, e.g., generated based on each player's point of view. Any numberof clients are local (e.g., co-located) or are geographically dispersed.The number of clients included in a game system vary widely from one ortwo to thousands, tens of thousands, or more. As used herein, the term“game player” is used to refer to a person that plays a game and theterm “game playing device” is used to refer to a device used to play agame. In some embodiments, the game playing device may refer to aplurality of computing devices that cooperate to deliver a gameexperience to the user.

For example, a game console and an HMD may cooperate with a video serversystem to deliver a game viewed through the HMD. In one embodiment, thegame console receives the video stream from the video server system andthe game console forwards the video stream, or updates to the videostream, to the HMD and/or television for rendering.

Still further, an HMD is used for viewing and/or interacting with anytype of content produced or used, such video game content, moviecontent, video clip content, web content, advertisement content, contestcontent, gamboling game content, conference call/meeting content, socialmedia content (e.g., posting, messages, media streams, friend eventsand/or game play), video portions and/or audio content, and content madefor consumption from sources over the internet via browsers andapplications and any type of streaming content. Of course, the foregoinglisting of content is not limiting, as any type of content can berendered so long as it can be viewed in the HMD or rendered to a screenor screen of the HMD.

Clients may, but are not required to, further include systems configuredfor modifying received video. For example, a client is configured toperform further rendering, to overlay one video image on another videoimage, to crop a video image, and/or the like. As another example,clients are configured to receive various types of video frames, such asI-frames, P-frames and B-frames, and to process these frames into imagesfor display to a user. In some embodiments, a member of clients isconfigured to perform further rendering, shading, conversion to 3-D,conversion to 2D, distortion removal, sizing, or like operations on thevideo stream. A member of clients is optionally configured to receivemore than one audio or video stream.

Input devices of clients includes, for example, a one-hand gamecontroller, a two-hand game controller, a gesture recognition system, agaze recognition system, a voice recognition system, a keyboard, ajoystick, a pointing device, a force feedback device, a motion and/orlocation sensing device, a mouse, a touch screen, a neural interface, acamera, input devices yet to be developed, and/or the like.

A video source includes rendering logic, e.g., hardware, firmware,and/or software stored on a computer readable medium such as storage.This rendering logic is configured to create video frames of the videostream based on the game state. All or part of the rendering logic isoptionally disposed within one or more graphics processing unit (GPU).Rendering logic typically includes processing stages configured fordetermining the three-dimensional spatial relationships between objectsand/or for applying appropriate textures, etc., based on the game stateand viewpoint. The rendering logic produces raw video that is encoded.For example, the raw video is encoded according to an Adobe Flash®standard, HTML-5, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv,Lagarith, MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, or the like.The encoding process produces a video stream that is optionally packagedfor delivery to a decoder on a device. The video stream is characterizedby a frame size and a frame rate. Typical frame sizes include 800×600,1280×720 (e.g., 720 p), 1024×768, 1080 p, although any other frame sizesmay be used. The frame rate is the number of video frames per second. Inone embodiment, a video stream includes different types of video frames.For example, the H.264 standard includes a “P” frame and a “I” frame.I-frames include information to refresh all macro blocks/pixels on adisplay device, while P-frames include information to refresh a subsetthereof. P-frames are typically smaller in data size than are I-frames.As used herein the term “frame size” is meant to refer to a number ofpixels within a frame. The term “frame data size” is used to refer to anumber of bytes required to store the frame.

In some embodiments, the client is a general purpose computer, a specialpurpose computer, a game console, a personal computer, a laptopcomputer, a tablet computer, a mobile computing device, a portablegaming device, a cellular phone, a set-top box, a streaming mediainterface/device, a smart television or networked display, or any othercomputing device capable of being configured to fulfill thefunctionality of a client as defined herein. In one embodiment, a cloudgaming server is configured to detect the type of client device which isbeing utilized by the user, and provide a cloud-gaming experienceappropriate to the user's client device. For example, image settings,audio settings and other types of settings may be optimized for theuser's client device.

FIG. 25 illustrates an embodiment of an Information Service Providerarchitecture. Information Service Providers (ISP) 2702 delivers amultitude of information services to users 2700-1, 2700-2, 2700-3,2700-4, etc., geographically dispersed and connected via the computernetwork 2210. In one embodiment, an ISP delivers one type of service,such as stock price updates, or a variety of services such as broadcastmedia, news, sports, gaming, etc. Additionally, the services offered byeach ISP are dynamic, that is, services can be added or taken away atany point in time. Thus, the ISP providing a particular type of serviceto a particular individual can change over time. For example, a user isserved by an ISP in near proximity to the user while the user is in herhome town, and the user is served by a different ISP when the usertravels to a different city. The home-town ISP will transfer therequired information and data to the new ISP, such that the userinformation “follows” the user to the new city making the data closer tothe user and easier to access. In another embodiment, a master-serverrelationship is established between a master ISP, which manages theinformation for the user, and a server ISP that interfaces directly withthe user under control from the master ISP. In another embodiment, thedata is transferred from one ISP to another ISP as the client movesaround the world to make the ISP in better position to service the userbe the one that delivers these services.

ISP 2702 includes Application Service Provider (ASP) 2706, whichprovides computer-based services to customers over the computer network2210. Software offered using an ASP model is also sometimes calledon-demand software or software as a service (SaaS). A simple form ofproviding access to a particular application program (such as customerrelationship management) is by using a standard protocol such as HTTP.The application software resides on the vendor's system and is accessedby users through a web browser using HTML, by special purpose clientsoftware provided by the vendor, or other remote interface such as athin client.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe computer network 2210. Users do not need to be an expert in thetechnology infrastructure in the “cloud” that supports them. In oneembodiment, cloud computing are divided in different services, such asInfrastructure as a Service (IaaS), Platform as a Service (PaaS), andSoftware as a Service (SaaS). Cloud computing services often providecommon business applications online that are accessed from a webbrowser, while the software and data are stored on the servers. The termcloud is used as a metaphor for the Internet (e.g., using servers,storage and logic), based on how the Internet is depicted in computernetwork diagrams and is an abstraction for the complex infrastructure itconceals.

Further, ISP 2702 includes a Game Processing Server (GPS) 2708 which isused by game clients to play single and multiplayer video games. Mostvideo games played over the Internet operate via a connection to a gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GPS establishes communication between the players andtheir respective game-playing devices exchange information withoutrelying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by the software company that owns thegame title, allowing them to control and update content.

Broadcast Processing Server (BPS) 2710 distributes audio or videosignals to an audience. Broadcasting to a very narrow range of audienceis sometimes called narrowcasting. The final leg of broadcastdistribution is how the signal gets to the listener or viewer, and itmay come over the air as with a radio station or TV station to anantenna and receiver, or may come through cable TV or cable radio (or“wireless cable”) via the station or directly from a network. TheInternet may also bring either radio or TV to the recipient, especiallywith multicasting allowing the signal and bandwidth to be shared.Historically, broadcasts have been delimited by a geographic region,such as national broadcasts or regional broadcast. However, with theproliferation of fast internet, broadcasts are not defined bygeographies as the content can reach almost any country in the world.

Storage Service Provider (SSP) 2712 provides computer storage space andrelated management services. SSPs also offer periodic backup andarchiving. By offering storage as a service, users can order morestorage as required. Another major advantage is that SSPs include backupservices and users will not lose all their data if their computers' harddrives fail. Further, in an embodiment, a plurality of SSPs have totalor partial copies of the user data, allowing users to access data in anefficient way independently of where the user is located or the devicebeing used to access the data. For example, a user can access personalfiles in the home computer, as well as in a mobile phone while the useris on the move.

Communications Provider 2714 provides connectivity to the users. Onekind of Communications Provider is an Internet Service Provider (ISP)which offers access to the Internet. The ISP connects its customersusing a data transmission technology appropriate for delivering InternetProtocol datagrams, such as dial-up, DSL, cable modem, fiber, wirelessor dedicated high-speed interconnects. The Communications Provider canalso provide messaging services, such as e-mail, instant messaging, andSMS texting. Another type of Communications Provider is the NetworkService provider (NSP) which sells bandwidth or network access byproviding direct backbone access to the Internet. Network serviceproviders, in one embodiment, include telecommunications companies, datacarriers, wireless communications providers, Internet service providers,cable television operators offering high-speed Internet access, etc.

Data Exchange 2704 interconnects the several modules inside ISP 2702 andconnects these modules to users 2700 via the computer network 2210. DataExchange 2704 covers a small area where all the modules of ISP 2702 arein close proximity, or covers a large geographic area when the differentmodules are geographically dispersed. For example, Data Exchange 2788includes a fast Gigabit Ethernet (or faster) within a cabinet of a datacenter, or an intercontinental virtual area network (VLAN).

Users 2700 access the remote services with client device 2720, whichincludes at least a CPU, a display and I/O. The client device can be aPC, a mobile phone, a netbook, tablet, gaming system, a PDA, etc. In oneembodiment, ISP 2702 recognizes the type of device used by the clientand adjusts the communication method employed. In other cases, clientdevices use a standard communications method, such as html, to accessISP 2702.

It should be noted that although some of the embodiments are describedherein with respect to a hand of the user 302, the embodiments applysimilarly to another body part of the user 302.

Embodiments described in the present disclosure may be practiced withvarious computer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theembodiments described in the present disclosure can also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a wire-based or wirelessnetwork.

With the above embodiments in mind, it should be understood that theembodiments described in the present disclosure can employ variouscomputer-implemented operations involving data stored in computersystems. These operations are those requiring physical manipulation ofphysical quantities. Any of the operations described herein that formpart of the embodiments described in the present disclosure are usefulmachine operations. Some embodiments described in the present disclosurealso relate to a device or an apparatus for performing these operations.The apparatus can be specially constructed for the required purpose, orthe apparatus can be a general-purpose computer selectively activated orconfigured by a computer program stored in the computer. In particular,various general-purpose machines can be used with computer programswritten in accordance with the teachings herein, or it may be moreconvenient to construct a more specialized apparatus to perform therequired operations.

Some embodiments described in the present disclosure can also beembodied as computer readable code on a computer readable medium. Thecomputer readable medium is any data storage device that can store data,which can be thereafter be read by a computer system. Examples of thecomputer readable medium include a hard drive, a NAS, a ROM, a RAM, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, an optical data storagedevice, a non-optical data storage device, etc. The computer readablemedium can include computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

It should be noted that in some embodiments, any of the embodimentsdescribed herein can be combined with any of the remaining embodiments.

Moreover, although some of the above-described embodiments are describedwith respect to a gaming environment, in some embodiments, instead of agame, other environments, e.g., a video conferencing environment, etc.,is used.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing embodiments described in the present disclosurehas been described in some detail for purposes of clarity ofunderstanding, it will be apparent that certain changes andmodifications can be practiced within the scope of the appended claims.Accordingly, the present embodiments are to be considered asillustrative and not restrictive, and the embodiments are not to belimited to the details given herein, but may be modified within thescope and equivalents of the appended claims.

The invention claimed is:
 1. A system for tracking, comprising: an inputdevice configured to be moved by a user, wherein the input deviceincludes a light emitter configured to emit light; a head-mounteddisplay (HMD) configured to be worn on a head of the user, wherein theHMD is configured to display a scene for a program during execution ofthe program, wherein the HMD has a camera configured to face the inputdevice to detect the light for capturing image data of the movement ofthe input device, wherein the HMD is configured to use the image data ofthe movement of the input device to facilitate a determination ofchanges in positions of the input device, wherein the HMD is configuredto display a movement of a virtual object in the scene based on thechanges in the positions of the input device; and an additional inputdevice configured to be moved by the user, wherein the additional inputdevice includes a light emitter configured to emit additional light,wherein the camera is configured to detect the additional light forcapturing additional image data of movement of the additional inputdevice, wherein the HMD is configured to use the additional image dataof the movement of the additional input device to facilitate adetermination of changes in positions of the additional input device,wherein the HMD is configured to display a movement of another virtualobject in the scene based on the changes in the positions of theadditional input device.
 2. The system of claim 1, wherein the inputdevice is configured to be worn on a hand of the user.
 3. The system ofclaim 1, wherein the camera is an infrared camera and the light emitteris an infrared light emitter.
 4. The system of claim 1, wherein thevirtual object is a virtual hand, wherein the HMD is configured todisplay the movement the virtual hand in the scene with movement of ahand of the user, wherein the input device is configured to be worn onthe hand.
 5. The system of claim 1, wherein the camera is located at abottom edge of the HMD, wherein the camera has a lens having afield-of-view in which the input device is located.
 6. The system ofclaim 1, wherein the HMD is configured to use the image data of themovement of the input device to facilitate a determination of changes inorientations of the input device, wherein the movement of the virtualobject in the scene is displayed based on the changes in theorientations of the input device.
 7. A system for tracking, comprising:an input device configured to be moved by a user; a head-mounted display(HMD) configured to be worn on a head of the user, wherein the HMD isconfigured to display a scene for a program during execution of theprogram, wherein the HMD has a camera configured to face the inputdevice to capture image data of the movement of the input device;wherein the HMD is configured to use the image data of the movement ofthe input device to facilitate a determination of changes in positionsof the input device, wherein the HMD is configured to display a movementof a virtual object in the scene based on the changes in the positionsof the input device; and an additional input device, wherein the camerais configured to capture additional image data of movement of theadditional input device, wherein the HMD is configured to use theadditional image data of the movement of the additional input device tofacilitate a determination of changes in positions of the additionalinput device, wherein the HMD is configured to display a movement ofanother virtual object in the scene based on the changes in thepositions of the additional input device.
 8. The system of claim 7,wherein the input device is configured to be worn on a hand of the user.9. The system of claim 7, wherein the virtual object is a virtual hand,wherein the HMD is configured to move the virtual hand in the scene withmovement of a hand of the user, wherein the input device is configuredto be worn on the hand.
 10. The system of claim 7, wherein the camera islocated at a bottom edge of the HMD, wherein the camera has a lenshaving a field-of-view in which the input device is located.
 11. Thesystem of claim 7, wherein the image data of the movement of the inputdevice is sent to facilitate a determination of changes in orientationsof the input device, wherein the movement of the virtual object in thescene is displayed based on the changes in the orientations of the inputdevice.
 12. A method for tracking an input device, comprising:displaying by a head-mounted display (HMD), a scene for a program duringexecution of the program and during movement of the input device by auser, wherein the HMD is configured to be worn on the head of the user;capturing, by a camera of the HMD, image data of the movement of theinput device; sending the image data of the movement of the input deviceto facilitate a determination of changes in positions of the inputdevice; displaying, by the HMD, a movement of a virtual object in thescene based on the changes in the positions of the input device;detecting, by the camera of the HMD, an additional light emitted from anadditional input device when the additional input device is moved by theuser, wherein the additional light is detected to capture additionalimage data of movement of the additional input device; sending theadditional image data of the movement of the additional input device tofacilitate a determination of changes in positions of the additionalinput device; and displaying, by the HMD, a movement of another virtualobject in the scene based on the changes in the positions of theadditional input device.
 13. The method of claim 12, further comprisingemitting light from the input device configured to be moved by the user,wherein the light is detected by the camera for said capturing of theimage data regarding the movement of the input device.
 14. The method ofclaim 13, wherein the camera is an infrared camera and the light emittedin infrared.
 15. The method of claim 12, wherein the input device isconfigured to be worn on a hand of the user.
 16. The method of claim 12,wherein the virtual object is a virtual hand, wherein the virtual handmoves in the scene with movement of a hand of the user, wherein theinput device is configured to be worn on the hand.
 17. The method ofclaim 12, wherein the camera is located at a bottom edge of the HMD,wherein the camera has a lens having a field-of-view in which the inputdevice is located.
 18. The method of claim 12, wherein the image data ofthe movement of the input device facilitates a determination of changesin orientations of the input device, wherein the movement of the virtualobject in the scene is displayed based on the changes in theorientations of the input device.